Data is said to follow the transform (or analysis) sparsity model if it becomes sparse when acted on by a linear operator called a sparsifying transform. Several algorithms have been designed to learn such a transform directly from data, and data-adaptive sparsifying transforms have demonstrated excellent performance i…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “transformer learning”
Learning Generalized Transformation Equivariant Representations via Autoencoding Transformationscs.CV
Transformation Equivariant Representations (TERs) aim to capture the intrinsic visual structures that equivary to various transformations by expanding the notion of {\em translation} equivariance underlying the success of Convolutional Neural Networks (CNNs). For this purpose, we present both deterministic AutoEncoding…
Transformers can learn spectral methods and perform unsupervised learning.
problem Learning spectral methods using unsupervised learning.
method Using multi-layered Transformers, pre-trained on a large set of instances, to learn and perform statistical estimation tasks.
result Proven that pre-trained Transformers can learn spectral methods and perform tasks like PCA and clustering.
Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.
problem Understanding and handling non-stationary environments in reinforcement learning.
method Demonstrated that transformers can achieve nearly optimal dynamic regret bounds in non-stationary settings.
result Transformers can approximate and learn strategies for non-stationary environments, matching or outperforming existing expert algorithms.
Transform learning improves K-means clustering for document analysis.
problem Improving K-means clustering for document analysis.
method Embedding K-means clustering loss into transform learning framework and solving jointly using ADMM.
result Improves over state-of-the-art in document clustering.
Transformers learn to make decisions in new contexts from offline data.
problem Understanding when and how transformers can perform in-context reinforcement learning.
method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.
result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.
Data augmentation (DA) is fundamental against overfitting in large convolutional neural networks, especially with a limited training dataset. In images, DA is usually based on heuristic transformations, like geometric or color transformations. Instead of using predefined transformations, our work learns data augmentati…
Transformers solve Gaussian Mixture Models without supervision.
problem Solving Gaussian Mixture Models (GMMs) unsupervised.
method Proposes TGMM, a transformer-based framework for GMM tasks.
result Transformers can effectively solve GMM tasks, improving upon classical methods.
Continuum transformers learn operators in context via gradient descent.
problem Generalizing transformers to handle infinite-dimensional inputs for in-context learning.
method Gradient descent in an operator RKHS, leveraging generalized representer theorems and gradient flows.
result Operator learned in context is Bayes Optimal Predictor in infinite depth limit.
Transformers can learn noisy linear systems with depth and IID data.
problem Learning noisy linear dynamical systems with transformers.
method Theoretical analysis of multi-layer and single-layer transformers with respect to -testing loss.
result Single-layer transformers have a non-diminishing lower bound on approximation error, suggesting depth separation.
New analysis shows transformer models can't learn effectively.
problem Transformer models learning in context can't achieve general predictive accuracy.
method Empirical evidence and mathematical analysis of transformer architecture limitations.
result Transformers cannot achieve general predictive accuracy due to architectural limitations.
Transformers can learn optimal variable selection in group-sparse classification.
problem Understanding how transformers leverage attention to select relevant variables in group-sparse classification.
method Training a one-layer transformer using gradient descent to select variables from one group of input variables.
result A one-layer transformer can correctly leverage the attention mechanism to select variables, disregarding irrelevant ones.
Efficient privacy-preserving machine learning framework using random transformations.
problem Slow training and inference speed in privacy-preserving machine learning systems.
method Random transformations like linear and permutation, combined with arithmetic sharing.
result High efficiency and low computation cost in private machine learning.
Learned data models based on sparsity are widely used in signal processing and imaging applications. A variety of methods for learning synthesis dictionaries, sparsifying transforms, etc., have been proposed in recent years, often imposing useful structures or properties on the models. In this work, we focus on sparsif…
Strongly polynomial algorithm for approximate Forster transforms and halfspace learning.
problem Computing approximate Forster transforms and halfspace learning.
method Strongly polynomial time algorithm for approximate Forster transforms and halfspace learning.
result First strongly polynomial time algorithm for distribution-free PAC learning of halfspaces.
Proposes a new framework for learning image augmentations to improve classification performance.
problem Improving classification performance with a given class of predictors.
method Transformed Risk Minimization (TRM) framework that optimizes both predictive models and data transformations.
result Performance of TRM with SCALE algorithm compares favorably to prior methods on CIFAR10/100.
PatchGT uses non-trainable graph patches to improve graph representation learning.
problem Learning high-level information in graph tasks with direct Transformer models.
method PatchGT segments graphs into non-trainable patches, uses GNN for patch-level learning, and Transformer for graph-level learning.
result PatchGT achieves higher expressiveness and competitive performance on benchmark datasets.
Fast linear transforms are ubiquitous in machine learning, including the discrete Fourier transform, discrete cosine transform, and other structured transformations such as convolutions. All of these transforms can be represented by dense matrix-vector multiplication, yet each has a specialized and highly efficient (su…
Deep learning model estimates uncertainty in complex regression tasks.
problem Uncertainty quantification in probabilistic regression predictions.
method Combines statistical and deep learning transformation models using gradient descent.
result State-of-the-art performance on small datasets and complex image data.
Our goal is to extract meaningful transformations from raw images, such as varying the thickness of lines in handwriting or the lighting in a portrait. We propose an unsupervised approach to learn such transformations by attempting to reconstruct an image from a linear combination of transformations of its nearest neig…
Transformer learns long-term dependencies from real-world data.
problem Sample inefficiency in deep reinforcement learning.
method Transformer architecture applied to autoregressive real-world episodes.
result Transformer-based world model generates meaningful experience.
A new model improves CT image quality from low-dose scans.
problem Improving CT image quality from low-dose scans.
method Multi-layer Residual Sparsifying Transform (MRST) learning model for low-dose CT reconstruction.
result The MRST model outperforms conventional methods in maintaining subtle details.
Transformers learn causal structure through gradient descent on self-attention mechanisms.
problem Understanding how transformers learn causal structure during training.
method In-context learning task and simplified two-layer transformer model.
result Gradient descent on a simplified transformer learns to encode latent causal graphs.
Transformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Modelsstat.ML
Transformers learn to play games in-context, proving Nash equilibrium.
problem Understanding in-context game-playing capabilities of pre-trained transformers.
method Theoretical guarantees and constructional results for transformer architecture in multi-agent games.
result Pre-trained transformers can learn Nash equilibrium in-context for two-player zero-sum games.
This paper studies transformer learning dynamics and initialization.
problem Understanding how transformers learn Markov chains and the role of initialization.
method First-order Markov chains and single-layer transformers, proving learning dynamics and conditions for convergence.
result Transformer parameters can converge to global or local minima based on initialization and Markovian data properties.
New method learns identity-preserving transformations on data manifolds without labels.
problem Learning identity-preserving transformations on natural variations without supervision.
method Introduces a learning strategy that does not require transformation labels and learns local regions for operators.
result Trains on MNIST and Fashion MNIST, and CelebA, learning transformations without labels.
The complexity of a learning task is increased by transformations in the input space that preserve class identity. Visual object recognition for example is affected by changes in viewpoint, scale, illumination or planar transformations. While drastically altering the visual appearance, these changes are orthogonal to r…
Paper proposes linear transformers for efficient in-context learning without context length limitations.
problem Quadratic complexity of softmax transformers limits data processing speed.
method Investigates linear transformers under domain generalization, showing they learn mappings from context distributions to response functions.
result Linear transformers achieve in-context learning with a linear complexity in context length, offering a dimension-independent convergence rate.
Looped Transformers improve robustness and expressivity in in-context learning for diverse tasks.
problem Improving robustness and expressivity in in-context learning for diverse tasks.
method Study in-context linear regression with diverse tasks, focusing on depth and looping.
result Looped Transformers exhibit similar expressive power and are provably robust under mild assumptions.
Transformer learns context and regularization for ICL in inverse problems.
problem Learning context and effective regularization for transformer-based in-context learning (ICL) in inverse problems.
method Introduced a linear transformer to learn inverse mapping from contextual examples to weight vectors, addressing rank-deficient problems.
result Transformer implicitly learns a prior distribution and effective regularization strategy, outperforming traditional methods.
Transformers can learn optimal regression mixtures efficiently.
problem Limited adoption of tailored regression methods due to their model-specific nature.
method Constructed a generative process for a mixture of linear regressions and used transformers to learn optimal predictors.
result Transformers achieve low mean-squared error and make predictions close to the optimal procedure.
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descentcs.LG
Transformers learn to perform logistic regression in-context.
problem Understanding how transformers learn to perform specific tasks in-context.
method Constructed multi-layer transformers that perform in-context logistic regression through normalized gradient descent.
result Transformers can be trained to perform in-context logistic regression effectively.
Learning invariant representations is an important problem in machine learning and pattern recognition. In this paper, we present a novel framework of transformation-invariant feature learning by incorporating linear transformations into the feature learning algorithms. For example, we present the transformation-invari…
This paper precisely estimates transformer derivatives for explicit learning guarantees.
problem Computing fully-explicit generalization bounds for transformers with precise higher-order derivative estimates.
method Analyzes and estimates all higher-order derivatives of transformers with multiple attention heads and layer normalization.
result Obtains explicit pathwise generalization bounds for transformers learning from non-i.i.d. samples.
Transformers learn algorithms for in-context learning with bounds and stability analysis.
problem Understanding and formalizing in-context learning as an algorithm learning problem.
method Formalizing ICL as a multitask learning problem, deriving generalization bounds, and analyzing stability.
result Transformers can implement near-optimal algorithms for classical regression tasks with i.i.d. and dynamic data.
Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
Paper presents a Transformer model for automatic domain adaptation.
problem Challenges in selecting or designing domain adaptation algorithms.
method Transformer model approximates and selects domain adaptation algorithms.
result Transformers can approximate and automatically select domain adaptation algorithms.
The wavelet transform has seen success when incorporated into neural network architectures, such as in wavelet scattering networks. More recently, it has been shown that the dual-tree complex wavelet transform can provide better representations than the standard transform. With this in mind, we extend our previous meth…
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
Transformers learn to integrate information from past positions incrementally, specializing heads in distinct patterns.
problem How transformers learn to integrate information from multiple past positions with varying statistical significance.
method High-order Markov chain task, incremental learning, sparse attention patterns, simplified differential equations, stage-wise convergence, early stopping as regularizer.
result Transformers learn to specialize heads in distinct patterns, shifting from competitive to cooperative learning dynamics.
We prove Transformers can learn diverse Gröbner bases.
problem Training Transformers for Gröbner basis computation.
method Prove generality of dataset generation algorithm; propose extended algorithm.
result Datasets are sufficiently general for diverse Gröbner bases learning.
New unsupervised learning technique learns independent kernels for better machine learning tasks.
problem Improving unsupervised representation learning for machine learning tasks.
method Stacking convolutional transforms using alternating proximal minimization scheme.
result DCTL outperforms shallow version CTL on benchmark datasets.
Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncs.LG
Transformers can adaptively select and perform various machine learning tasks in context.
problem Transformers' ability to perform multiple machine learning tasks without explicit prompting.
method Statistical theory and efficient implementation of in-context gradient descent, enabling transformers to implement various algorithms and tasks.
result Transformers can implement a broad class of standard machine learning algorithms in context, including algorithm selection.
Transformers can approximate Newton's method for logistic regression.
problem Implementing higher order optimization methods in Transformers.
method Linear attention Transformers with ReLU layers approximating second order optimization algorithms.
result Transformers can implement a single step of Newton's iteration for matrix inversion.
Many applications in signal processing benefit from the sparsity of signals in a certain transform domain or dictionary. Synthesis sparsifying dictionaries that are directly adapted to data have been popular in applications such as image denoising, inpainting, and medical image reconstruction. In this work, we focus in…
New approach learns image transformations directly for clustering.
problem Learning better deep representations for image clustering.
method Directly learns transformations and clusters in image space without abstract features.
result Jointly learns prototypes and transformations using deep learning modules.
Looped transformers with LN converge to power method for principal component prediction.
problem Understanding how transformers learn algorithmic procedures.
method Study of principal component prediction with looped linear transformers and layer normalization.
result Gradient descent trains looped transformers with LN to implement the power method for principal component prediction.