Survey categorizes attention models across various domains.
problem Understanding and improving attention models in neural networks.
method Taxonomy and review of existing techniques.
result Provides a structured overview of attention models.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,982 papers · 148 categories
Trend · papers per month
48 results for “neural attentive”
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
Mathematical framework for understanding attention in neural networks.
problem Lack of theoretical understanding of attention in neural networks.
method Proposes a measure-theoretic model of attention and interprets self-attention as a system of self-interacting particles.
result Shows that attention is Lipschitz-continuous under suitable assumptions.
Aligns attention distributions for improved accuracy and robustness.
problem Improving the accuracy and robustness of neural networks using attention mechanisms.
method Alignment attention that encourages key and query distributions to match within each head.
result Alignment attention leads to better accuracy, uncertainty estimation, and robustness across various tasks.
RANP improves neural processes for sequential data.
problem Capturing temporal order and recurrent structure from sequential data.
method Incorporated ANP into a recurrent neural network.
result RANP outperforms NPs and LSTMs in 1D regression and autonomous-driving tasks.
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
Graph neural networks benefit from attention under specific conditions.
problem Understanding and improving the effectiveness of attention in graph neural networks.
method Designing controlled graph reasoning tasks, analyzing performance under various conditions, proposing weakly-supervised training.
result Attention can provide significant gains in performance under certain conditions, but its effect is often negligible or harmful.
Attentive Neural Controlled Differential Equations for Time-series Classification and Forecastingcs.LG
ANCDEs improve time-series forecasting and classification using attention in NCDEs.
problem Improving time-series forecasting and classification using neural controlled differential equations.
method Integrating attention into neural controlled differential equations (ANCDEs).
result ANCDEs consistently show the best accuracy in time-series classification and forecasting.
New neural network improves audio classification accuracy.
problem Challenging audio classification problem in pattern recognition.
method Introduces a Classifier-Attention-Based Convolutional Neural Network (CAB-CNN) with an attention mechanism to reduce classifier complexity.
result Significantly improves audio classification performance, achieving more than 10% improvements.
AttViz offers online visualizations of neural language model attention mechanisms.
problem Limited interpretability of neural language models.
method Online toolkit for exploring self-attention mechanisms.
result Visualizations help understand model decision-making.
Bayesian attention improves model performance and robustness.
problem Limited exploration of stochastic attention in neural networks.
method Introduces Bayesian attention belief networks using gamma and Weibull distributions.
result Outperforms deterministic and stochastic attention methods in accuracy and robustness.
NAS model improves social recommendation accuracy using neural attention.
problem Capturing and weighing friends' preferences in social recommendation systems.
method Proposes a Neural Attention mechanism (NAS) for Social collaborative filtering.
result NAS model outperforms state-of-the-art methods in publicly available datasets.
Neural nets predict user attention from mouse movements.
problem Predicting user attention from mouse cursor movements.
method Investigated different mouse movement representations and trained neural networks.
result Neural networks outperform handcrafted features for predicting user attention.
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property …
Models use embeddings and attention for better claim severity prediction.
problem Improving predictive modeling of claim severity with categorical data.
method Developed neural networks and attention-based architectures with embeddings.
result Attention-based models enhance predictive performance with contextual augmentation.
Linear attention model outperforms full attention in language tasks.
problem Quadratic scaling of full attention limits model size.
method Introduced a linear attention mechanism.
result Linear attention models achieve comparable performance to full attention models.
Deep neural network improves NILM with attention mechanism.
problem Energy disaggregation of individual appliance power demands from aggregate meter readings.
method Regression and classification subnetworks with attention mechanism.
result Proposed model outperforms state-of-the-art on REDD and UK-DALE datasets.
Lipschitz normalization boosts deep attention models, especially for graph neural networks.
problem Gradient explosion in deep graph attention networks leads to poor performance.
method Enforcing Lipschitz continuity by normalizing attention scores.
result Deep GAT models with LipschitzNorm achieve state-of-the-art results for tasks with long-range dependencies.
Enhanced GNN with expanded attention window and partially random embeddings.
problem Limited expressivity of traditional GNNs in distinguishing non-isomorphic graphs.
method Graph attention network with expanding attention window and partially random initial embeddings. Head dropout for regularization.
result Improved ability to differentiate between non-isomorphic graphs.
New attention mechanism improves meta-transfer learning in dynamic tasks.
problem Underfitting in meta-transfer learning with dynamic tasks.
method Proposed Recurrent Memory Reconstruction (RMR) attention mechanism.
result ASNP-RMR significantly outperforms baselines in various tasks.
IAL uses interactive learning to improve model performance with minimal human feedback.
problem Overfitting and high human interaction cost in training neural networks.
method IAL framework with NAP attention generator and reranking algorithm.
result IAL significantly outperforms baselines with less retraining and human interaction.
Deep neural network models have recently draw lots of attention, as it consistently produce impressive results in many computer vision tasks such as image classification, object detection, etc. However, interpreting such model and show the reason why it performs quite well becomes a challenging question. In this paper,…
Enhances neural rendering with geometry-aware attention.
problem Efficiently modeling complex 3D scenes.
method Introduces Epipolar Cross Attention (ECA) for non-local operations.
result Significant improvement in Generative Query Networks (GQN) performance.
Novel GNN model tackles few-shot learning with improved performance.
problem Few-shot learning with GNN suffers from over-fitting and over-smoothing.
method Proposes Attentive GNN with triple-attention mechanism.
result Improves GNN performance for few-shot learning tasks.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
MAGNA improves graph neural networks by incorporating multi-hop context information.
problem Limited context in current graph neural networks.
method Diffuses attention scores across the network, accounting for all paths between nodes.
result State-of-the-art performance on node classification and knowledge graph completion benchmarks.
CPA models improve GNNs by preserving node cardinality.
problem Limited understanding of attention-based GNNs' discriminative power.
method Theoretical analysis and CPA models to preserve cardinality information.
result CPA models can improve GNNs' performance in node and graph classification.
Memory-augmented neural networks improve machine translation performance.
problem Improving machine translation accuracy and flexibility.
method Evaluation of Neural Turing Machines and Differentiable Neural Computers for machine translation tasks.
result Memory-augmented neural networks perform similarly to attentional encoders on Vietnamese to English tasks but have lower BLEU scores on Romanian to English tasks.
Recently, graph neural networks have attracted great attention and achieved prominent performance in various research fields. Most of those algorithms have assumed pairwise relationships of objects of interest. However, in many real applications, the relationships between objects are in higher-order, beyond a pairwise …
Differentiable Window improves attention modules by enabling more focused attentions.
problem Improving attention mechanisms in neural networks.
method Proposes Differentiable Window, a neural module for dynamic window selection.
result Consistent and sizable improvements across various NLP tasks.
A new feature selection method using attention for neural networks.
problem Feature selection for neural networks with a budget constraint.
method Sequential Attention: greedy forward selection with attention weights.
result Achieves state-of-the-art results for neural networks.
A new multi-layer attention mechanism improves speech keyword recognition accuracy.
problem Inaccurate attention weights in LSTM networks for speech keyword recognition.
method Introducing information from layers prior to feature extraction into attention weights calculations.
result The proposed multi-layer attention mechanism leads to more accurate attention weights and improved keyword spotting performance.
AI2V learns user representations by focusing on recent interests.
problem User interests and behavior change over time, affecting recommendation quality.
method Introduces AI2V, a neural attentive model that learns user representations by focusing on recent interests.
result AI2V outperforms other models on various datasets.
SANs estimate feature importance in neural models, identifying interactions and high-ranked features.
problem Understanding and interpreting black-box neural network models.
method Self-Attention Network (SAN) architecture for feature importance estimation.
result SANs identify similar high-ranked features and feature interactions as other methods, improving predictive performance.
A new neural network learns from acoustic scenes by suppressing irrelevant patterns.
problem Acoustic scenes are rich and redundant, making classification challenging.
method Spatio-temporal attention pooling layer coupled with a convolutional recurrent neural network.
result The method outperforms a strong convolutional neural network baseline and sets new state-of-the-art performance.
Paper proposes a faster neural machine translation model using election methods and Q-learning.
problem Slower inference time in complex attention models.
method Modelled attention network using election methods and Q-learning.
result Inference time is less than a standard Bahdanau translator, results comparable.
Paper presents a neural network for recognizing human activities from unlabeled sensor data.
problem Time-consuming annotation of sensor data for activity recognition.
method Attention-based convolutional neural network for weakly labeled data.
result Attention model improves accuracy in recognizing human activities.
TNP-KR improves scalability of NPs with Transformer blocks and attention mechanisms.
problem Scalability bottleneck in NPs and GPs, especially for large datasets.
method Introduces TNP-KR with KRBlock, kernel-based attention, and two attention mechanisms.
result TNP-KR with DKA outperforms Performer and achieves state-of-the-art results.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
Attention mechanisms in deep learning become Gaussian process-like as the number of heads increases.
problem Understanding the behavior of attention mechanisms in deep learning models.
method Extending the equivalence between wide neural networks and Gaussian processes to attention architectures.
result Multi-head attention architectures behave as Gaussian processes as the number of heads tends to infinity.
Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layerscs.LG
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimiz…
PiNet improves graph classification efficiency and accuracy.
problem Graph level classification challenges.
method Attention-based pooling mechanism for graph convolution operations.
result Superior performance and high sample efficiency.
Study reveals self-attention's role in learning and generalizing interactions.
problem Understanding self-attention's theoretical role in neural architectures.
method Interacting entities analysis, including multi-agent RL and genetic sequences.
result Self-attention efficiently represents, learns, and generalizes pairwise interactions.
DAFT models attention as a dynamical system to make neural networks more interpretable.
problem Uninterpretable features learned by neural networks without human priors.
method DAFT models attention as a continuous dynamical system using neural ODEs.
result DAFT reduces the number of reasoning steps while maintaining similar performance.
The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past cell state values for a limited window size . The formulas ar…
Gaussian Attention Model and Its Application to Knowledge Base Embedding and Question Answeringstat.ML
We propose the Gaussian attention model for content-based neural memory access. With the proposed attention model, a neural network has the additional degree of freedom to control the focus of its attention from a laser sharp attention to a broad attention. It is applicable whenever we can assume that the distance in t…