Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under L∞-norm and Lebesgue integrable functions under Lp-norm. Multi-head attention outperforms single-head in in-context linear regression tasks.
problem Comparing performance of transformer with single-/multi-head attention in in-context learning.
method Theoretical analysis of performance of transformers with different attention mechanisms in linear regression tasks.
result Multi-head attention with a substantial embedding dimension outperforms single-head attention in in-context linear regression tasks.
Attention models can overfit without harming test performance.
problem Understanding benign overfitting in single-head attention models.
method Analyzing conditions for benign overfitting in a single-head softmax attention model.
result A single-head attention model can overfit without harming test performance under certain conditions.
Minimalist softmax attention learns constrained Boolean functions with supervision.
problem Learning k-bit Boolean functions, especially AND and OR. method Minimalist single-head softmax-attention mechanism with teacher forcing.
result Minimalist attention can solve Boolean tasks with supervision, but not without.
The study reveals the spectral structure of attention layers and its implications for generalization.
problem Understanding the spectral structure and generalization of trained attention layers.
method Empirical risk minimization in a single-head tied-attention layer, using random matrix theory, spin-glass theory, and approximate message passing.
result Exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and spectrum of the key and query matrices.
New theory shows how multi-head attention reduces variance and decorrelates outputs.
problem Understanding and optimizing multi-head attention in neural networks.
method Developed a statistical theory linking multi-head attention to ensemble Nadaraya-Watson estimators.
result MHA variance reduction depends on head decorrelation, not just head count.
Investigates the benefits of multi-head attention in Transformers, deriving convergence and generalization guarantees.
problem Underexplored dynamics of multi-head attention in Transformer training and generalization.
method Derives convergence and generalization guarantees for gradient-descent training of a multi-head self-attention model.
result Establishes conditions for initialization that ensure multi-head attention's realizability.
Single-head transformers with a single self-attention layer can approximate any sequence-to-sequence function and are efficient under certain conditions.
problem Statistical and computational limits of prompt tuning for transformer-based models.
method Investigation of single-head transformers with a single self-attention layer, proving universality and efficiency under SETH.
result Existence of almost-linear time prompt tuning inference algorithms under certain conditions.
New approach improves multi-head attention by making heads less similar.
problem Multi-head attention can lead to similar features, reducing model expressiveness.
method Proposes a non-parametric approach using Bayesian techniques to make heads repel each other.
result Improves feature diversity, leading to better representations and performance.
Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences, it retrieves richer information than a single-head attention which only summarizes the whole sequen…
Attention mechanisms in deep learning become Gaussian process-like as the number of heads increases.
problem Understanding the behavior of attention mechanisms in deep learning models.
method Extending the equivalence between wide neural networks and Gaussian processes to attention architectures.
result Multi-head attention architectures behave as Gaussian processes as the number of heads tends to infinity.
Bidirectional attention is shown to be equivalent to a continuous bag of words model with mixture-of-experts.
problem Understanding the statistical underpinnings of bidirectional attention.
method Exploring bidirectional attention as a mixture-of-experts model and reparameterizing it.
result Bidirectional attention can be viewed as a continuous bag of words model with mixture-of-experts weights.
HKT improves sequence processing with multi-scale attention and kernel analysis.
problem Processing sequences at multiple scales with efficient attention mechanisms.
method Trainable causal downsampling and convex weights for level-specific score matrices.
result HKT achieves consistent gains over standard attention across various tasks.
We demystify attention patterns in multi-head softmax models for linear data.
problem Understanding the training dynamics and emergent patterns in multi-head softmax attention models.
method Extensive empirical experiments and rigorous theoretical analysis.
result Multi-head softmax attention models approximate a debiased gradient descent predictor, outperforming single-head attention and achieving near-Bayesian optimality.
Study on multi-head softmax attention dynamics for in-context learning.
problem Understanding and optimizing multi-head softmax attention models for multi-task linear regression.
method Gradient flow analysis and spectral mapping technique.
result Gradient flow converges to optimal multi-head softmax attention model, with task allocation emerging during training.
Study shows the number of attention heads affects transformer performance.
problem Understanding how the number of attention heads impacts transformer performance.
method Introduced a generalized D-retrieval task, established upper and lower bounds on parameter complexity, and validated with experiments. result Transformers with many heads can efficiently approximate functions, while few heads require a large number of parameters.
Transformers can emulate various algorithms by prompting, proving universality.
problem How to emulate algorithms using fixed-weight Transformers.
method Two modes of in-context algorithm emulation: task-specific and prompt-programmable. Constructing prompts that encode algorithm parameters into token representations.
result Fixed-weight Transformers can emulate a broad class of algorithms via prompts.
This work relaxes energy constraints in self-attention layers for a more general analysis.
problem Understanding inherent biases and dynamics in self-attention layers without energy functions.
method Dynamical systems analysis and Jacobian matrix examination.
result Normalized dynamics are close to a critical state, indicating high inference performance.
Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic speech recognition (ASR) system into a single neural network. In previous work, we have shown that such architectures are comparable to stat…
Transformers can learn Markov processes with constant depth, surprising results.
problem Understanding how transformers learn context in Markov processes.
method Empirical study and theoretical analysis of attention-based transformers on Markov data.
result Transformers with constant depth can achieve low test loss on Markov sequences, matching empirical and theoretical findings.
Reformulates binary classification on manifolds using Yang-Mills-Higgs theory.
problem Binary classification on non-contractible spaces.
method Formulates binary classification as a Yang-Mills-Higgs variational problem, encoding data as a functor.
result Reveals a geometric interpretation of binary classification and solves XOR on the torus.
There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector…
Transformers without skip connections collapse token representations to a single direction.
problem Rapid convergence of token representations to a single direction in self-attention-only Transformers.
method Analysis of layer normalization, residual connections, and multi-head attention mechanisms.
result Residual connections prevent rank collapse in real Transformers, while MLPs generate new feature directions.
Nuclear segmentation in histology images is a challenging task due to significant variations in the shape and appearance of nuclei. One of the main hurdles in nuclear instance segmentation is overlapping nuclei where a smart algorithm is needed to separate each nucleus. In this paper, we introduce a proposal-free deep …
Sequential learning of tasks using gradient descent leads to an unremitting decline in the accuracy of tasks for which training data is no longer available, termed catastrophic forgetting. Generative models have been explored as a means to approximate the distribution of old tasks and bypass storage of real data. Here …
New interpretation of attention in Transformers and Graph Attention Networks.
problem Understanding and improving attention mechanisms in deep learning models.
method Decomposed attention into a kernel and a normalization term; generalized the kernel function and norm.
result Generalized attention leads to better performance on various tasks.
Regularizes attention scores in vision transformers using bootstrapping.
problem Noisy and diffused attention maps in ViT limit interpretability.
method Statistical learning techniques, bootstrapping of attention scores.
result Improves shrinkage and sparsity of attention scores.
Random forests with attention and self-attention improve regression performance.
problem Improving regression model performance on various datasets.
method Proposes new models using attention and self-attention mechanisms to solve regression problems.
result The models improve model performance on many datasets.
Aligns attention distributions for improved accuracy and robustness.
problem Improving the accuracy and robustness of neural networks using attention mechanisms.
method Alignment attention that encourages key and query distributions to match within each head.
result Alignment attention leads to better accuracy, uncertainty estimation, and robustness across various tasks.
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
problem Transformer models suffer from representation collapse and are vulnerable to contaminated samples.
method Uses Mahalanobis distance to define hyper-ellipsoidal neighborhoods for attention weights.
result Elliptical Attention reduces representation collapse and enhances model robustness.
Kernel PCA explains self-attention mechanisms in deep learning models.
problem Understanding and explaining self-attention mechanisms in deep learning models.
method Deriving self-attention from kernel principal component analysis (kernel PCA).
result RPC-Attention, a robust attention mechanism, outperforms softmax attention in various tasks.
Study examines asset pricing using various attention models, finding global self-attention and sliding window sparse attention models perform well.
problem Traditional asset pricing models miss temporal dependency and short memory issues.
method Investigates RNN attention models with various attention mechanisms for large-cap US stocks.
result Global self-attention and sliding window sparse attention models outperform in deriving returns and hedging risks, especially during the pandemic.
LARF improves random forests with attention mechanisms and contamination models.
problem Improving accuracy in classification tasks with random forests.
method Introduces a two-level attention mechanism and uses a mixture of contamination models.
result Significantly improved classification performance on various datasets.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
S3Attention improves long sequence attention with smoothed skeleton sketching.
problem Quadratic complexity of vanilla Attention makes it unsuitable for long sequence tasks.
method S3Attention uses smoothing and matrix sketching to balance information preservation and computation. result S3Attention significantly outperforms vanilla Attention and other Attention variants. Mathematical framework for understanding attention in neural networks.
problem Lack of theoretical understanding of attention in neural networks.
method Proposes a measure-theoretic model of attention and interprets self-attention as a system of self-interacting particles.
result Shows that attention is Lipschitz-continuous under suitable assumptions.
Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent …
GOAT improves attention mechanisms by learning better priors.
problem Standard attention mechanisms use a naive uniform prior, limiting flexibility and generalization.
method GOAT introduces a trainable, continuous prior that replaces the uniform assumption, maintaining compatibility with optimized kernels.
result GOAT avoids representational trade-offs and learns an extrapolatable prior that combines positional flexibility with length generalization.
Bayesian attention improves model performance and robustness.
problem Limited exploration of stochastic attention in neural networks.
method Introduces Bayesian attention belief networks using gamma and Weibull distributions.
result Outperforms deterministic and stochastic attention methods in accuracy and robustness.
Improved language models with talking-heads attention.
problem Language model perplexity and quality issues.
method Added linear projections in multi-head attention.
result Better perplexities and quality in language tasks.
SympFormer accelerates attention blocks using inertial dynamics on density spaces.
problem Improving the efficiency of self-attention blocks in Transformers.
method Introduced accelerated attention blocks derived from inertial Nesterov dynamics on density spaces.
result Accelerated attention blocks converge faster than classical blocks while preserving oracle calls.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
Bayesian theory explains abrupt emergence of copy subcircuit in attention.
problem Understanding the abrupt emergence of the copy subcircuit in attention during training.
method Deriving a closed-form posterior over the attention matrix and reducing it to a low-dimensional order parameter space.
result Derive a phase transition in the amount of training data.
Attention weights may not accurately highlight important parts due to combinatorial shortcuts.
problem Inaccurate interpretation of attention weights in models.
method Theoretical analysis and design of experiments to show combinatorial shortcuts. Proposed two methods to mitigate this issue.
result Proposed methods improve the interpretability of attention mechanisms.
Improved graph attention model for noisy graphs.
problem Understanding and improving graph attention in noisy graphs.
method Proposes SuperGAT, a self-supervised graph attention network.
result SuperGAT learns more expressive attention by encoding edges.
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.