A new method for estimating sparse inverse covariance matrices.
problem Recovering the connectivity and non-connectivity graph of covariates.
method Adaptive thresholding in a transformed domain of the inverse covariance matrix.
result The proposed method outperforms state-of-the-art methods in accuracy.
Paper presents a Transformer model for automatic domain adaptation.
problem Challenges in selecting or designing domain adaptation algorithms.
method Transformer model approximates and selects domain adaptation algorithms.
result Transformers can approximate and automatically select domain adaptation algorithms.
Many applications in signal processing benefit from the sparsity of signals in a certain transform domain or dictionary. Synthesis sparsifying dictionaries that are directly adapted to data have been popular in applications such as image denoising, inpainting, and medical image reconstruction. In this work, we focus in…
Although modern recommendation systems can exploit the structure in users' item feedback, most are powerless in the face of new users who provide no structure for them to exploit. In this paper we introduce ImplicitCE, an algorithm for recommending items to new users during their sign-up flow. ImplicitCE works by trans…
Paper proposes linear transformers for efficient in-context learning without context length limitations.
problem Quadratic complexity of softmax transformers limits data processing speed.
method Investigates linear transformers under domain generalization, showing they learn mappings from context distributions to response functions.
result Linear transformers achieve in-context learning with a linear complexity in context length, offering a dimension-independent convergence rate.
VCoTTA uses variational Bayesian methods to adapt models under continuous domain shifts.
problem Error accumulation in continual test-time adaptation.
method VCoTTA employs variational Bayesian techniques to update a Bayesian Neural Network (BNN) during testing, combining priors from source and teacher models.
result VCoTTA effectively mitigates error accumulation in CTTA, as shown by experimental results on three datasets.
This research improves representation learning for new domains with limited new supervision.
problem Learning representations that generalize well to new domains with minimal new data.
method Encourages linearity of factors of variation through learned linear transformations called latent canonicalizers.
result Reduces the number of observations needed to generalize to a similar target domain compared to supervised baselines.
Two-layer model sparsifies image residuals for CT image reconstruction.
problem Image reconstruction from limited and corrupted data.
method Pre-learning a two-layer sparsifying transform model with block coordinate descent optimization.
result Preliminary experiments show the two-layer model improves CT image reconstruction from low-dose measurements.
A new model improves CT image quality from low-dose scans.
problem Improving CT image quality from low-dose scans.
method Multi-layer Residual Sparsifying Transform (MRST) learning model for low-dose CT reconstruction.
result The MRST model outperforms conventional methods in maintaining subtle details.
Natural signals and images are well-known to be approximately sparse in transform domains such as Wavelets and DCT. This property has been heavily exploited in various applications in image processing and medical imaging. Compressed sensing exploits the sparsity of images or image patches in a transform domain or synth…
DECT-MULTRA improves material decomposition in CT images.
problem Noise and artifacts degrade material images in DECT imaging.
method Combines PWLS estimation with MULTRA model for efficient clustering and sparse coding.
result Superior material image quality and decomposition accuracy compared to other methods.
A modern Hopfield network improves deep learning with better memory and attention.
problem Improving memory and attention mechanisms in deep learning.
method Introducing a new Hopfield network with continuous states and a novel update rule.
result The new Hopfield network can store and retrieve patterns efficiently with low errors.
Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
Framework for safely updating machine learning models.
problem Continuous updates to machine learning models can lead to unintended consequences.
method Formalizes the problem as computing the largest locally invariant domain (LID), uses tractable primal-dual formulation.
result Matches or exceeds heuristic baselines for avoiding forgetting while providing formal safety guarantees.
MTL2L learns to adapt optimisation rules for unseen data.
problem Learners need to adapt to unseen data domains.
method Introduces MTL2L, a context-aware neural optimiser.
result MTL2L can adapt optimisation rules for unseen data.
New algorithms update dynamic graph regression faster than existing methods.
problem Efficiently updating linear regression solutions for dynamic graphs.
method Subsampled randomized Hadamard transform and CountSketch.
result First sublinear update time randomized algorithms for dynamic graph regression.
Introduces Wasserstein Transform for updating distance structures.
problem Enhancing features and denoising data sets.
method Representing data points as probability measures and updating distances via Wasserstein distance.
result GT is computationally cheaper and has a closed-form solution for ℓ2-Wasserstein distance. New algorithm learns deterministic policies in continuous domains using sign of advantage function.
problem Learning deterministic policies in continuous domains.
method Proposes a new trust region algorithm, Penalized NFAC (PeNFAC), based on a theoretical explanation of the advantage function.
result PeNFAC algorithm surpasses state-of-the-art algorithms in learning deterministic policies in classic control problems.
Transform learning improves MRI image reconstruction from sparse data.
problem Efficiently reconstruct MRI images from limited data.
method TL-based methods using learned models and transform domains.
result TL-based methods outperform classical CS methods in MRI reconstruction.
Transformer models show robustness across domains with domain adversarial training.
problem Domain adaptation from multiple sources with no labeled data.
method Domain adversarial training and mixture of experts.
result Domain adversarial training improves representation but not performance.
Pion optimizes LLMs by preserving weight matrix singular values.
problem Training large language models (LLMs) with standard optimizers leads to unstable weight matrices.
method Pion uses orthogonal transformations to update weight matrices, preserving their singular values.
result Pion offers a stable alternative to standard optimizers for LLM pretraining and finetuning.
Learning multiple tasks across heterogeneous domains is a challenging problem since the feature space may not be the same for different tasks. We assume the data in multiple tasks are generated from a latent common domain via sparse domain transforms and propose a latent probit model (LPM) to jointly learn the domain t…
Proves nonemptyness of domains for specific group actions.
problem Nonemptyness of domains of proper discontinuity for Anosov groups of affine Lorentzian transformations.
method Proof of nonemptyness of domains of proper discontinuity.
result Proves nonemptyness of domains for Anosov groups of affine Lorentzian transformations.
This work improves Fourier pricing for multi-asset options using RQMC with domain transformation.
problem Efficiently pricing multi-asset options in high dimensions with Fourier methods.
method Randomized quasi-Monte Carlo (RQMC) with domain transformation to handle singularities.
result RQMC with domain transformation provides accurate and scalable Fourier pricing for multi-asset options.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
AI updates can disrupt human-AI teams; new objective improves compatibility.
problem AI updates can disrupt human-AI team performance.
method Introduce compatibility concept, propose re-training objective.
result Current machine learning algorithms do not produce compatible updates.
2-simplicial Transformer enhances logical reasoning in reinforcement learning.
problem Logical reasoning in reinforcement learning.
method Introduces 2-simplicial Transformer with higher-dimensional attention and tensor product updates. result Shows effectiveness of 2-simplicial Transformer for logical reasoning. Test-time training adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts.
problem Improving accuracy of pretrained models under distribution shifts.
method Explaining TTT behavior through a decision-theoretic lens.
result TTT reduces prediction error when updates are spectrally matched to the prompt's signal-to-noise ratio and aligned with query-relevant eigen-directions.
Tent adapts models during testing by minimizing entropy of predictions.
problem Adapting models to new data during testing with limited information.
method Test entropy minimization (tent) and online channel-wise affine transformations.
result Reduces generalization error on various datasets and benchmarks.
The characteristics (or numerical patterns) of a feature vector in the transform domain of a perturbation model differ significantly from those of its corresponding feature vector in the input domain. These differences - caused by the perturbation techniques used for the transformation of feature patterns - degrade the…
Fewer data weight updates lead to faster convergence in machine learning models.
problem Improving robustness of machine learning models through data mixing.
method Analyzing convergence behavior of data mixing with a finite number of inner steps.
result The optimal number of inner steps scales with the budget and type of gradients used.
New method eliminates domain size restrictions for X-ray transform inversion.
problem Injectivity and stability of X-ray transform in convex domains.
method Semiclassical analysis to invert X-ray transform without small domain assumptions.
result Elimination of domain size restrictions for injectivity and stability.
PCGS-TF uses a Transformer to adaptively control expert switching in non-stationary environments.
problem Static regret is insufficient for strictly online prediction in non-stationary settings.
method Policy-Controlled Generalized Share (PCGS) with a Transformer as an update controller.
result PCGS-TF achieves the lowest dynamic regret in non-stationary families and expert pools.
GENIE balances domain-invariant feature learning and gradient alignment for improved DG performance.
problem Domain Generalization (DG) overfitting to domain-specific features
method GENIE (Generalization-ENhancing Iterative Equalizer) optimizer
result Prevents a small subset of parameters from dominating optimization, promoting domain-invariant feature learning
Transformers adapted to spherical geometry using space-filling curves.
problem Generalizing transformers to geometric domains like spheres.
method Attention heads following a space-filling curve.
result Introduction of the Spiroformer on a 2-sphere.
Transformers enable in-context learning with guarantees for a wide range of tasks.
problem How to enable in-context learning with transformers for various tasks.
method Developed a universal approximation theory integrating Barron's function approximation with transformer capabilities.
result Transformers can approximate any target function with vanishingly small risk using a few in-context examples.
Method uses JPEG transform for faster image classification.
problem Efficient image classification with compressed data.
method Reformulates residual networks for JPEG compressed images.
result Mathematically equivalent to spatial domain networks up to ReLu approximation.
We present an updated version of the VESICLE-CNN algorithm presented by Roncal et al. (2014). The original implementation makes use of a patch-based approach. This methodology is known to be slow due to repeated computations. We update this implementation to be fully convolutional through the use of dilated convolution…
OLinear forecasts time series more efficiently by transforming data orthogonally.
problem Efficiently forecasting time series with entangled dependencies.
method OLinear uses OrthoTrans to transform data orthogonally, then applies NormLin for linear layer.
result OLinear achieves state-of-the-art performance with high efficiency.
Extends SW and GSW to compare heterogeneous joint distributions.
problem Limited applicability of SW and GSW to heterogeneous joint distributions.
method Introduces HHRT and PGRT to extend SW and GSW.
result H2SW distance for heterogeneous joint distributions.
We show how to convert ICL in linearized transformers into model weights.
problem Making in-context learning interpretable and permanent in large language models.
method Demonstrates equivalence between ICL and bias terms in linearized transformers, and develops ICLCA for exact conversion.
result Exact conversion of in-context learning into model weights is possible for linearized transformers.
Functions with constant geodesic X-ray transform are restricted to manifolds with specific geometrical properties.
problem Existence of functions with constant geodesic X-ray transform on manifolds.
method Analyzing the geometrical properties of manifolds based on the existence of such functions.
result Functions with constant geodesic X-ray transform impose specific geometrical restrictions on the manifold.
This work tackles OOD generalization by leveraging causal invariance without needing to recover causal features.
problem Learning models that perform well on out-of-distribution (OOD) data.
method Causal invariant transformations to modify non-causal features while preserving causal parts.
result Theoretical and practical methods to learn a minimax optimal model across domains using single domain data.
Stability is a key aspect of data analysis. In many applications, the natural notion of stability is geometric, as illustrated for example in computer vision. Scattering transforms construct deep convolutional representations which are certified stable to input deformations. This stability to deformations can be interp…
TTT improves transformer models for in-context learning.
problem Improving transformer models for efficient in-context learning.
method Gradient-based TTT method for linear transformers, with theoretical and empirical analysis.
result TTT significantly reduces the sample size required for in-context learning.
BOIL updates model body only, showing better few-shot learning performance.
problem Few-shot learning efficiency with model reuse vs. change.
method Proposes BOIL, updating only model body, freezing head.
result Significantly outperforms MAML on cross-domain tasks.
Transforms game optimization dynamics into frequency domain for precise hyperparameter analysis.
problem Analyzing convergence of hyperparameters in game optimization.
method Frequency-domain framework using High-Resolution Differential Equations (HRDEs) and Laplace transforms.
result Derives precise convergence criteria for the Lookahead algorithm.
This paper explores estimating chaotic dynamics and parameters using local ensemble Kalman filters.
problem Estimating chaotic dynamics and parameters from observations.
method Local ensemble Kalman filters with covariance and local domain localisation.
result Rigorously updating global parameters using a local domain ensemble Kalman filter.