A method to automatically choose feature dimensions in linear attention for better approximation quality.
problem Choosing the feature dimension in linear attention to balance quality and efficiency.
method Statistical degrees of freedom for determining feature dimension, layer-wise training strategy.
result Our method achieves smaller approximation error compared to fixed dimensions and improves model performance.
Efficiently accelerates attention calculation for Transformers with relative positional encoding.
problem Quadratic complexity of attention in long sequences.
method Kernelized attention with Fast Fourier Transform (FFT) for RPE.
result Achieves O(n log n) time complexity, mitigates training instability, and outperforms other models.
Linformer reduces transformer complexity to linear, improving efficiency.
problem High cost of training and deploying large transformer models for long sequences.
method Approximates self-attention with low-rank matrix, proposing Linformer with O(n) complexity. result Linformer performs similarly to standard transformers but is more memory- and time-efficient.
SpGAT learns graph representations using spectral attention for efficiency.
problem Efficiently capturing global graph patterns with minimal parameters.
method Introduces Spectral Graph Attention Network (SpGAT) using spectral domain attention mechanisms and a fast Chebychev approximation.
result SpGAT achieves better global pattern recognition with fewer parameters compared to GAT.
A new LSTM architecture improves time series forecasting efficiency.
problem Efficiency and accuracy in time series forecasting using linear models.
method Attention-free LSTM architecture for time series prediction.
result Improved prediction capacity and efficiency compared to LSTM.
ZeroS improves Transformers by adding negative weights, matching or beating softmax attention.
problem Limited performance of linear attention methods, especially in long context sequences.
method Proposes Zero-Sum Linear Attention (ZeroS) that removes the zero-order term and reweights zero-sum softmax residuals.
result ZeroS matches or exceeds standard softmax attention across various benchmarks, theoretically expanding representable functions.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
Efficient sparse attention reduces self-attention complexity and improves model performance.
problem Quadratic compute and memory requirements of self-attention for long sequences.
method Content-based sparse attention with dynamic routing module.
result Routing Transformer achieves state-of-the-art performance on various benchmarks.
PiNet improves graph classification efficiency and accuracy.
problem Graph level classification challenges.
method Attention-based pooling mechanism for graph convolution operations.
result Superior performance and high sample efficiency.
Transformers with linear space and time complexity for accurate attention estimation.
problem Efficiently estimating attention in large-scale tasks without relying on priors.
method Performers use Fast Attention Via positive Orthogonal Random features (FAVOR+) for linear approximation of softmax attention.
result Performers achieve competitive results on various tasks, demonstrating the effectiveness of their attention-learning approach.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.
problem Transformer sample inefficiency in simple tasks.
method Demonstrated through parity-learning setup, showing CoT reduces required samples from exponential to polynomial.
result Transformer learns function within polynomial samples with CoT, requiring exponential samples without CoT.
Abstractor enhances Transformers for relational reasoning, improving sample efficiency and performance.
problem Improving sample efficiency and performance in relational tasks.
method Introduces Abstractor module with relational cross-attention to enable explicit relational reasoning.
result Dramatic improvements in sample efficiency and performance on various relational tasks.
Study shows attention-style models learn pairwise interactions efficiently.
problem Learning pairwise interactions in attention-style models.
method Proved minimax rate of convergence for learning pairwise interactions.
result Minimax rate is M−2β+12β independent of embedding dimension and token number. Express improves causal attention guarantees for language models.
problem Improving causal attention guarantees for language models.
method Introducing Express, a tool for converting non-causal attention into causal with matching guarantees.
result Express improves causal attention guarantees to log^(3/2)(n)/s with minimal memory and compression overhead.
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
A new model uses attention and Gaussian processes for efficient time-series generation.
problem Computational inefficiency and uncertainty underestimation in sequence transduction.
method Attention-based Gaussian process network for real-valued sequence generation.
result The model improves training efficiency and learns factorized generative distribution.
This paper analyzes the interpolation error of nonlinear Attention compared to linear regression.
problem Understanding the interpolation error of nonlinear Attention in high-dimensional settings.
method Derives explicit expressions for mean-squared interpolation error using signal-plus-noise model and random matrix theory.
result Nonlinear Attention generally incurs a larger interpolation error than linear regression, but this gap can be reversed with structured signals.
A new algorithm reduces memory usage for long token attention in streaming applications.
problem Memory inefficiency in computing attention for long documents.
method One-pass streaming algorithm using sublinear space storage.
result Super-efficient memory usage for long token attention.
A brain-inspired spiking Transformer reduces energy consumption and enhances interpretability.
problem Energy inefficiency and lack of interpretability in Transformer models.
method Spiking STDP Transformer using spike-timing-dependent plasticity (STDP) for self-attention.
result Achieves 94.35% and 78.08% accuracy on CIFAR-10 and CIFAR-100 datasets respectively, with 88.47% energy reduction.
A-MMSE uses attention to learn efficient OFDM channel estimation.
problem Accurate OFDM channel estimation requires second-order statistics, which are hard to obtain in practice.
method A-MMSE is a model-based DNN framework that learns linear MMSE filters via Attention Transformer, reducing inference complexity.
result A-MMSE outperforms other methods in normalized MSE across various SNR conditions.
Paper proposes ADC framework to reduce ViT SL training communication overhead.
problem Reducing communication overhead in ViT SL training.
method Two parallel compression strategies: class-agnostic merging and token discarding.
result Significantly reduces communication overhead without sacrificing accuracy.
Paper extends sparse alternatives to softmax for continuous domains, enabling efficient attention mechanisms.
problem Efficiently assigning zero probability to irrelevant categories in continuous domains.
method Extend alpha-entmax to continuous domains, introducing continuous-domain attention mechanisms.
result Continuous attention allows attending to time intervals and compact regions, improving text classification, machine translation, and visual question answering.
Visual attention serves as a means of feature selection mechanism in the perceptual system. Motivated by Broadbent's leaky filter model of selective attention, we evaluate how such mechanism could be implemented and affect the learning process of deep reinforcement learning. We visualize and analyze the feature maps of…
The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate wh…
ConViT combines CNN and ViT strengths, improving image classification.
problem Combining the strengths of CNNs and ViTs while avoiding their limitations.
method Introducing GPSA, a form of positional self-attention with a soft convolutional inductive bias.
result ConViT outperforms DeiT on ImageNet while offering improved sample efficiency.
Transformer-MGK replaces redundant heads with Gaussian key mixtures, improving efficiency and performance.
problem Redundant attention heads in transformers degrade performance and efficiency.
method Transformer-MGK replaces redundant heads with a mixture of Gaussian keys.
result Transformer-MGK accelerates training and inference, reduces parameters and FLOPs, and achieves comparable or better accuracy.
This paper addresses the challenging task of video captioning which aims to generate descriptions for video data. Recently, the attention-based encoder-decoder structures have been widely used in video captioning. In existing literature, the attention weights are often built from the information of an individual modali…
LatFormer improves geometric reasoning by incorporating lattice symmetry priors in attention mechanisms.
problem State-of-the-art models struggle with geometric reasoning tasks in the ARC and LARC datasets.
method Introduced LatFormer, a model that uses lattice symmetry priors in attention masks.
result LatFormer requires 2 orders of magnitude fewer data than standard attention mechanisms.
Transformers learn functionals from distributions without losing information.
problem Lack of rigorous mathematical theory supporting Transformer performance.
method Proposed a Transformer learning framework, attention operator, and distribution regression.
result Transformers can compress distributions into function representations without loss of information.
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
A new Hopfield model reduces inefficiency in large transformer models.
problem Inefficiency in training large transformer-based models.
method Introduces an Outlier-Efficient Hopfield Model (OutEffHop) to improve model performance.
result Achieves up to 26% reduction in model output norms across four models.
Paper proposes a new hierarchical attention mechanism for multi-scale data.
problem Challenges in applying neural attention to multi-scale, multi-modal data.
method Developed a mathematical framework for multi-modal, multi-scale data and derived optimal neural attention mechanics.
result Proposed hierarchical attention mechanism improves transformer performance in multi-scale, multi-modal settings.
This paper proposes an attention module augmented relational network called SARN(Sequential Attention Relational Network) that can carry out relational reasoning by extracting reference objects and making efficient pairing between objects. SARN greatly reduces the computational and memory requirements of the relational…
Crowdsourcing is a relatively economic and efficient solution to collect annotations from the crowd through online platforms. Answers collected from workers with different expertise may be noisy and unreliable, and the quality of annotated data needs to be further maintained. Various solutions have been attempted to ob…
TNP-KR improves scalability of NPs with Transformer blocks and attention mechanisms.
problem Scalability bottleneck in NPs and GPs, especially for large datasets.
method Introduces TNP-KR with KRBlock, kernel-based attention, and two attention mechanisms.
result TNP-KR with DKA outperforms Performer and achieves state-of-the-art results.
A study on optimizing self-attention in tabular data using Optimal Transport.
problem Improving efficiency and accuracy of self-attention in tabular classification tasks.
method Developed an OT-based algorithm to generate class-specific dummy Gaussian distributions and train an MLP.
result Achieved comparable accuracy to Transformers with reduced computational cost and efficiency.
CauScale efficiently discovers causal relationships in large graphs.
problem Efficiency bottlenecks in causal discovery for large graphs.
method Neural architecture with reduction unit and tied attention weights.
result Achieves 99.6% mAP on in-distribution data and 84.4% on out-of-distribution data.
Minimum attention improves reinforcement learning performance in high-dimensional dynamics.
problem Improving reinforcement learning performance in high-dimensional nonlinear dynamics.
method Applying minimum attention as a regularization technique in reinforcement learning, including model-based and model-free approaches.
result Minimum attention outperforms state-of-the-art algorithms in few-shot adaptation and variance reduction.
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptiv…
Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.
problem Vanishing gradients in recurrent networks when capturing long-term dependencies.
method Formal analysis of self-attention's effect on gradient propagation, proposing a relevancy screening mechanism.
result Self-attention mitigates vanishing gradients in recurrent networks, providing guarantees.
Linear attention in Transformers can be interpreted as dynamic VAR models.
problem Misalignment between Transformers and autoregressive forecasting objectives.
method Interpreting linear attention as VAR, rearranging MLP, attention, and flow.
result SAMoVAR improves performance, interpretability, and efficiency.
Improves sample efficiency in reinforcement learning with input representation.
problem Poor sample efficiency in reinforcement learning.
method Attention-based method to project inputs into an invariant representation space.
result Representation space is m! smaller for inputs of m objects, improving sample efficiency. IAL uses interactive learning to improve model performance with minimal human feedback.
problem Overfitting and high human interaction cost in training neural networks.
method IAL framework with NAP attention generator and reranking algorithm.
result IAL significantly outperforms baselines with less retraining and human interaction.
Clustered attention improves transformer efficiency for large sequences.
problem Quadratic complexity of transformer attention matrix for large sequences.
method Group queries into clusters, compute attention only for centroids, and use centroids to approximate key/query dot products.
result Linear complexity with respect to sequence length for a fixed number of clusters.
Improves point-cloud reconstruction by optimizing projections with self-attention.
problem Inefficient and non-metric projection methods for sliced Wasserstein distances.
method Proposes distributional sliced Wasserstein distance with self-attention for permutation-invariant and metric optimization.
result Self-attention amortized distributional projection optimization achieves better performance in point-cloud reconstruction.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
Neural nets predict user attention from mouse movements.
problem Predicting user attention from mouse cursor movements.
method Investigated different mouse movement representations and trained neural networks.
result Neural networks outperform handcrafted features for predicting user attention.