A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent …
Mathematical framework for understanding attention in neural networks.
problem Lack of theoretical understanding of attention in neural networks.
method Proposes a measure-theoretic model of attention and interprets self-attention as a system of self-interacting particles.
result Shows that attention is Lipschitz-continuous under suitable assumptions.
Aligns attention distributions for improved accuracy and robustness.
problem Improving the accuracy and robustness of neural networks using attention mechanisms.
method Alignment attention that encourages key and query distributions to match within each head.
result Alignment attention leads to better accuracy, uncertainty estimation, and robustness across various tasks.
RANP improves neural processes for sequential data.
problem Capturing temporal order and recurrent structure from sequential data.
method Incorporated ANP into a recurrent neural network.
result RANP outperforms NPs and LSTMs in 1D regression and autonomous-driving tasks.
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
ANCDEs improve time-series forecasting and classification using attention in NCDEs.
problem Improving time-series forecasting and classification using neural controlled differential equations.
method Integrating attention into neural controlled differential equations (ANCDEs).
result ANCDEs consistently show the best accuracy in time-series classification and forecasting.
New neural network improves audio classification accuracy.
problem Challenging audio classification problem in pattern recognition.
method Introduces a Classifier-Attention-Based Convolutional Neural Network (CAB-CNN) with an attention mechanism to reduce classifier complexity.
result Significantly improves audio classification performance, achieving more than 10% improvements.
AttViz offers online visualizations of neural language model attention mechanisms.
problem Limited interpretability of neural language models.
method Online toolkit for exploring self-attention mechanisms.
result Visualizations help understand model decision-making.
Bayesian attention improves model performance and robustness.
problem Limited exploration of stochastic attention in neural networks.
method Introduces Bayesian attention belief networks using gamma and Weibull distributions.
result Outperforms deterministic and stochastic attention methods in accuracy and robustness.
Neural nets predict user attention from mouse movements.
problem Predicting user attention from mouse cursor movements.
method Investigated different mouse movement representations and trained neural networks.
result Neural networks outperform handcrafted features for predicting user attention.
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property …
Models use embeddings and attention for better claim severity prediction.
problem Improving predictive modeling of claim severity with categorical data.
method Developed neural networks and attention-based architectures with embeddings.
result Attention-based models enhance predictive performance with contextual augmentation.
Deep neural network improves NILM with attention mechanism.
problem Energy disaggregation of individual appliance power demands from aggregate meter readings.
method Regression and classification subnetworks with attention mechanism.
result Proposed model outperforms state-of-the-art on REDD and UK-DALE datasets.
Lipschitz normalization boosts deep attention models, especially for graph neural networks.
problem Gradient explosion in deep graph attention networks leads to poor performance.
method Enforcing Lipschitz continuity by normalizing attention scores.
result Deep GAT models with LipschitzNorm achieve state-of-the-art results for tasks with long-range dependencies.
Enhanced GNN with expanded attention window and partially random embeddings.
problem Limited expressivity of traditional GNNs in distinguishing non-isomorphic graphs.
method Graph attention network with expanding attention window and partially random initial embeddings. Head dropout for regularization.
result Improved ability to differentiate between non-isomorphic graphs.
New attention mechanism improves meta-transfer learning in dynamic tasks.
problem Underfitting in meta-transfer learning with dynamic tasks.
method Proposed Recurrent Memory Reconstruction (RMR) attention mechanism.
result ASNP-RMR significantly outperforms baselines in various tasks.
Deep neural network models have recently draw lots of attention, as it consistently produce impressive results in many computer vision tasks such as image classification, object detection, etc. However, interpreting such model and show the reason why it performs quite well becomes a challenging question. In this paper,…
IAL uses interactive learning to improve model performance with minimal human feedback.
problem Overfitting and high human interaction cost in training neural networks.
method IAL framework with NAP attention generator and reranking algorithm.
result IAL significantly outperforms baselines with less retraining and human interaction.
Enhances neural rendering with geometry-aware attention.
problem Efficiently modeling complex 3D scenes.
method Introduces Epipolar Cross Attention (ECA) for non-local operations.
result Significant improvement in Generative Query Networks (GQN) performance.
Novel GNN model tackles few-shot learning with improved performance.
problem Few-shot learning with GNN suffers from over-fitting and over-smoothing.
method Proposes Attentive GNN with triple-attention mechanism.
result Improves GNN performance for few-shot learning tasks.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
MAGNA improves graph neural networks by incorporating multi-hop context information.
problem Limited context in current graph neural networks.
method Diffuses attention scores across the network, accounting for all paths between nodes.
result State-of-the-art performance on node classification and knowledge graph completion benchmarks.
Memory-augmented neural networks improve machine translation performance.
problem Improving machine translation accuracy and flexibility.
method Evaluation of Neural Turing Machines and Differentiable Neural Computers for machine translation tasks.
result Memory-augmented neural networks perform similarly to attentional encoders on Vietnamese to English tasks but have lower BLEU scores on Romanian to English tasks.
Recently, graph neural networks have attracted great attention and achieved prominent performance in various research fields. Most of those algorithms have assumed pairwise relationships of objects of interest. However, in many real applications, the relationships between objects are in higher-order, beyond a pairwise …
Differentiable Window improves attention modules by enabling more focused attentions.
problem Improving attention mechanisms in neural networks.
method Proposes Differentiable Window, a neural module for dynamic window selection.
result Consistent and sizable improvements across various NLP tasks.
A new feature selection method using attention for neural networks.
problem Feature selection for neural networks with a budget constraint.
method Sequential Attention: greedy forward selection with attention weights.
result Achieves state-of-the-art results for neural networks.
As an important part of speech recognition technology, automatic speech keyword recognition has been intensively studied in recent years. Such technology becomes especially pivotal under situations with limited infrastructures and computational resources, such as voice command recognition in vehicles and robot interact…
AI2V learns user representations by focusing on recent interests.
problem User interests and behavior change over time, affecting recommendation quality.
method Introduces AI2V, a neural attentive model that learns user representations by focusing on recent interests.
result AI2V outperforms other models on various datasets.
SANs estimate feature importance in neural models, identifying interactions and high-ranked features.
problem Understanding and interpreting black-box neural network models.
method Self-Attention Network (SAN) architecture for feature importance estimation.
result SANs identify similar high-ranked features and feature interactions as other methods, improving predictive performance.
Paper proposes a faster neural machine translation model using election methods and Q-learning.
problem Slower inference time in complex attention models.
method Modelled attention network using election methods and Q-learning.
result Inference time is less than a standard Bahdanau translator, results comparable.
TNP-KR improves scalability of NPs with Transformer blocks and attention mechanisms.
problem Scalability bottleneck in NPs and GPs, especially for large datasets.
method Introduces TNP-KR with KRBlock, kernel-based attention, and two attention mechanisms.
result TNP-KR with DKA outperforms Performer and achieves state-of-the-art results.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
Attention mechanisms in deep learning become Gaussian process-like as the number of heads increases.
problem Understanding the behavior of attention mechanisms in deep learning models.
method Extending the equivalence between wide neural networks and Gaussian processes to attention architectures.
result Multi-head attention architectures behave as Gaussian processes as the number of heads tends to infinity.
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimiz…
PiNet improves graph classification efficiency and accuracy.
problem Graph level classification challenges.
method Attention-based pooling mechanism for graph convolution operations.
result Superior performance and high sample efficiency.
Study reveals self-attention's role in learning and generalizing interactions.
problem Understanding self-attention's theoretical role in neural architectures.
method Interacting entities analysis, including multi-agent RL and genetic sequences.
result Self-attention efficiently represents, learns, and generalizes pairwise interactions.
The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past cell state values for a limited window size k. The formulas ar…
We propose the Gaussian attention model for content-based neural memory access. With the proposed attention model, a neural network has the additional degree of freedom to control the focus of its attention from a laser sharp attention to a broad attention. It is applicable whenever we can assume that the distance in t…
We have successfully implemented the "Learn to Pay Attention" model of attention mechanism in convolutional neural networks, and have replicated the results of the original paper in the categories of image classification and fine-grained recognition.
New approach shows why overparameterized neural nets generalize well.
problem Understanding why overparameterized neural nets generalize well in practice.
method An alternative notion of capacity for attention-based models based on the effective rank of attention matrices.
result Generalization bound matches empirical scaling laws observed in large language models.
A new neural network separates vocals from music accompaniment.
problem Separating vocals from music accompaniment in recordings.
method Self-attention convolutional neural network (CNN) with densely-connected blocks.
result 19.5% relative improvement in vocals separation.
Pyramid Attention Networks improve image restoration by leveraging self-similarities across scales.
problem Lack of full exploitation of self-similarities in image restoration by recent deep learning methods.
method Introduces a Pyramid Attention module that processes multi-scale feature correspondences to borrow clean signals from coarser levels.
result Pyramid Attention module achieves state-of-the-art results in various image restoration tasks.
In this paper, we present a novel approach for incorporating external knowledge in Recurrent Neural Networks (RNNs). We propose the integration of lexicon features into the self-attention mechanism of RNN-based architectures. This form of conditioning on the attention distribution, enforces the contribution of the most…
The paper explores how to make neural networks extrapolate longer sequences.
problem Neural networks struggle to extrapolate beyond seen data, especially for longer sequences.
method The authors propose a model with separate content- and location-based attention mechanisms.
result Models with the proposed attention mechanisms are better at extrapolating longer sequences.
Improved neural transducer model outperforms attention model on longer sequences.
problem Improving performance of neural transducer models.
method Comparison of training criteria (marginalization vs. maximum approximation), model generalization, and output label topology.
result Final transducer model outperforms attention model by over 6% relative WER on Switchboard 300h.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.