arXiv research

One-layer transformers can't solve induction heads task efficiently.

This study analyzes how one-layer transformers learn regular language recognition tasks.

Transformers can learn optimal variable selection in group-sparse classification.

This paper compares Transformers and RNNs in various tasks, showing size differences.

Lower bounds set for infinite-precision transformers.

This paper explores how Transformers predict next tokens in autoregressive tasks.

Randomly initialized neural networks can linearly separate arbitrary sets.

This paper investigates how transformers can learn to generalize to unseen examples in context.

Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.

The study characterizes sets for which one-layer neural networks are positive.

Study explores how neural networks and Transformers learn modular arithmetic with multiple inputs.

LLMs can be tricked into recalling facts based on context clues.

Transformers struggle to learn Markovian dynamics, showing NP-hard optimization challenges.

Gradient descent fails to learn simple neural networks efficiently.

Law derived for neural networks with sparse connections.

Transformers learn random walks optimally with gradient descent.

One LN layer stabilizes neural network extrapolation.

Generative adversarial networks (GANs) are a widely used framework for learning generative models. Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax optimization problem to global optimality, but are in practice successfully trained using stochastic gradient descent-ascent.…

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we fir…

Transformers solve parity problems efficiently with step-by-step reasoning.

Minimalistic unsupervised learning with sparse manifold transform achieves SOTA performance.

Transformer models perform slower than convolutional networks in learning hierarchical language structures.

Transformers learn multi-step reasoning through gradient descent.

Softmax attention approximates complex functions and subsumes many known universal approximators.

The paper analyzes the training dynamics of a transformer for next-token prediction.

Transformers learn unseen tasks via prompts without fine-tuning.

Transformers learn sparse Boolean functions through RL and SFT, revealing distinct learning behaviors.

Stability of non-abelian X-ray transform proven in higher dimensions.

The study estimates the expressiveness of GCNs with bounds on the number of linear regions.

Algorithm learns polynomial transformations of Gaussian distributions.

Transformers learn a mesa-optimizer to implement in-context learning.

Theoretical analysis of vision transformers' performance with MAE and CL objectives.

RNNs solve modular addition tasks using low rank and sparse Fourier structures.

It is often hypothesized that a crucial role for recurrent connections in the brain is to constrain the set of possible response patterns, thereby shaping the neural code. This implies the existence of neural codes that cannot arise solely from feedforward processing. We set out to find such codes in the context of one…

We introduce a new method for training deep Boltzmann machines jointly. Prior methods require an initial learning pass that trains the deep Boltzmann machine greedily, one layer at a time, or do not perform well on classifi- cation tasks.

Layer-wise networks have a closed-form solution and a stopping criterion.

Layer-wise networks have a closed-form solution and a stopping criterion.

We present a general framework for training deep neural networks without backpropagation. This substantially decreases training time and also allows for construction of deep networks with many sorts of learners, including networks whose layers are defined by functions that are not easily differentiated, like decision t…

Study on calibration and consistency of adversarial surrogate losses.

Study shows efficient neural network approach for stochastic bandits.

Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.

An efficient way to learn deep density models that have many layers of latent variables is to learn one layer at a time using a model that has only one layer of latent variables. After learning each layer, samples from the posterior distributions for that layer are used as training data for learning the next layer. Thi…

FLoE adapts LLMs by selectively deploying LoRA adapters based on layer importance and task requirements.

New approach to deeper graph neural networks to avoid performance degradation.

GIBLy adds geometric priors to 3D segmentation models, improving performance with minimal overhead.

Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum $\ell_2$ norm ("ridgeless") interpolation in high-dimensional least squares regression. …

Gradient flow on softmax attention minimizes nuclear norm of weight matrices.

The success of deep neural networks has inspired many to wonder whether other learners could benefit from deep, layered architectures. We present a general framework called forward thinking for deep learning that generalizes the architectural flexibility and sophistication of deep neural networks while also allowing fo…