Improved GAN performance with a novel local attention mechanism.
problem Enhancing the performance of Generative Adversarial Networks (GANs).
method Introducing a two-dimensional local attention mechanism that preserves geometry and locality, and using information flow graphs for design.
result Significant improvements in FID and Inception scores, from 18.65 to 15.94 on ImageNet.
Self-attention networks localize when eigenspectrum variance is small.
problem Self-attention mechanisms can lead to rank and entropy collapses, reducing model expressivity and trainability.
method Characterized attention localization using query-key eigenspectrum variance.
result Small eigenspectrum variance prevents both rank and entropy collapses, improving model performance.
AGNN improves network localization accuracy by 37-53% in NLOS conditions.
problem Massive network localization under Non-Line-of-Sight conditions.
method Attentional Graph Neural Network (AGNN) with Adjacency Learning Module (ALM) and Multiple Graph Attention Layers (MGAL).
result Significant improvement in localization accuracy, approaching fundamental lower bounds.
WAVE improves time series forecasting by integrating AR and MA components.
problem Time series forecasting challenges.
method WAVE attention mechanism with AR and MA components.
result WAVE attention consistently improves TSF performance.
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose an interpretable and easy plug-in spatial-temporal attention mechanism for video action recognition. For spatial attention, we learn a saliency mask to allow the model to focus…
GAT-AGNN learns stock trends using graph and attention mechanisms.
problem Predicting dynamic stock trends in a complex market.
method Sequential graph structure with attention mechanisms.
result GAT-AGNN outperforms state-of-the-art methods in stock trend prediction.
ETC improves Transformer models for long and structured inputs.
problem Scaling input length and encoding structured inputs in Transformers.
method Introduces global-local attention, relative position encodings, and CPC pre-training.
result Achieves state-of-the-art results on four natural language datasets.
Study combines CNNs and LSTMs for ECG classification, improving performance with attention mechanisms.
problem Improving ECG classification accuracy for weakly supervised tasks.
method Investigates ConvLSTM networks and various attention mechanisms to capture both morphological and temporal features.
result Attention mechanisms improve classification performance and provide interpretable saliency maps.
FedACS uses attention to select clients with similar data for federated learning.
problem Non-IID data and data scarcity in federated learning.
method FedACS integrates an attention mechanism to prioritize clients with similar data distributions.
result FedACS improves federated learning performance by addressing non-IID data and data scarcity.
Introduces recency bias to improve time-series forecasting.
problem Lack of recency bias in standard Transformer attention for time-series data.
method Reweights attention scores with a smooth heavy-tailed decay to emphasize nearby observations.
result Recency-biased attention consistently improves sequential modeling and achieves competitive performance on time-series forecasting benchmarks.
The paper identifies patterns in language model weights used for memorizing paragraphs.
problem Locating the specific mechanisms and weights used by language models to memorize paragraphs.
method Examined gradients and attention patterns in language models to identify memorized paragraphs.
result Gradients of memorized paragraphs have a distinguishable spatial pattern, and localized attention heads are involved in paragraph memorization.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
Active-memory mechanisms can replace self-attention in Transformers, but optimal results often require both.
problem Replacing self-attention with active-memory mechanisms in Transformers.
method Evaluation of various active-memory mechanisms in a Transformer model.
result Active-memory mechanisms can achieve comparable results to self-attention for language modeling, but optimal results are often achieved by combining both mechanisms.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
LNUCB-TA improves MAB performance by dynamically adjusting exploration rates and recognizing spatiotemporal patterns.
problem Suboptimal performance in environments with rapidly changing reward structures and static exploration rates.
method Hybrid model combining linear and nonlinear estimation, with adaptive k-NN for temporal attention.
result Significantly outperforms state-of-the-art algorithms in cumulative and mean reward, convergence, and robustness.
Study reveals attention mechanism's similarity computation parallels traditional machine learning.
problem Understanding the essence and principles of attention mechanism in deep learning.
method Examined classic metrics and vector space properties in manifold learning, clustering, and supervised learning to identify key characteristics of similarity computation and information propagation.
result Self-attention mechanism in deep learning adheres to the same principles but operates more flexibly and adaptively.
Differentiable Window improves attention modules by enabling more focused attentions.
problem Improving attention mechanisms in neural networks.
method Proposes Differentiable Window, a neural module for dynamic window selection.
result Consistent and sizable improvements across various NLP tasks.
Attention mechanism is a hot spot in deep learning field. Using channel attention model is an effective method for improving the performance of the convolutional neural network. Squeeze-and-Excitation block takes advantage of the channel dependence, selectively emphasizing the important channels and compressing the rel…
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
GOAT improves attention mechanisms by learning better priors.
problem Standard attention mechanisms use a naive uniform prior, limiting flexibility and generalization.
method GOAT introduces a trainable, continuous prior that replaces the uniform assumption, maintaining compatibility with optimized kernels.
result GOAT avoids representational trade-offs and learns an extrapolatable prior that combines positional flexibility with length generalization.
CoulGAT interprets GAT models by analyzing node interactions.
problem Understanding and interpreting the complex interactions within graph attention networks.
method Developed a CoulGAT framework to analyze and interpret GAT model layers and datasets.
result Extracted node-node and node-feature interactions to define a standard model for graph structure.
New technique improves imitation learning by preventing local minima and exploring states.
problem Behavioral cloning gets stuck in local minima and lacks effective exploration.
method Two-phase model with sampling mechanisms and self-attention modules.
result Significantly outperforms previous state-of-the-art in various environments.
Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalit…
Intelligent Momentum Transformer outperforms traditional trading strategies.
problem Improving time-series momentum and mean-reversion trading strategies.
method Attention-based deep-learning architecture (Momentum Transformer) combining attention and LSTM.
result Momentum Transformer outperforms benchmarks and adapts to new market regimes.
The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…
Neural HMM with AGA captures multi-scale dynamics in financial markets.
problem Capturing multi-scale temporal dynamics in financial markets.
method Parallel multi-resolution encoders, adaptive gating, and multi-head attention.
result Outperforms fixed-resolution baselines in predicting price movements and liquidity shocks.
Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local…
Topological attention improves forecasting of univariate time series.
problem Forecasting univariate time series using local topological features.
method Topological attention mechanism that integrates local topological properties into forecasting models.
result Topological attention leads to state-of-the-art performance on the M4 benchmark.
Paper tackles rDR classification and lesion segmentation using self-supervised equivariant learning and attention-based MIL.
problem Classifying rDR and segmenting lesions from image-level labels.
method Integrates self-supervised equivariant attention mechanism (SEAM) with attention-based multi-instance learning (MIL).
result Achieved AU ROC of 0.958 on Eyepacs dataset, outperforming state-of-the-art.
Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under L∞-norm and Lebesgue integrable functions under Lp-norm. KGRL uses reinforcement learning with knowledge graphs for better interactive recommendation.
problem Achieving responsiveness and accuracy in dynamic user-item interactions.
method KGRL combines reinforcement learning and knowledge graphs, using a local knowledge network and attention mechanism.
result KGRL outperforms state-of-the-art methods in simulated and real-world environments.
Paper proposes a new hierarchical attention mechanism for multi-scale data.
problem Challenges in applying neural attention to multi-scale, multi-modal data.
method Developed a mathematical framework for multi-modal, multi-scale data and derived optimal neural attention mechanics.
result Proposed hierarchical attention mechanism improves transformer performance in multi-scale, multi-modal settings.
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.
Tiled Squeeze-and-Excite improves channel attention with local spatial context.
problem Improving channel attention mechanisms in neural networks.
method Proposes tiled squeeze-and-excite (TSE) framework for channel attention.
result Local context of 7 rows or columns is sufficient for matching global context performance.
CRAUM-Net improves salient object detection with context and uncertainty modeling.
problem Accurate salient object detection with precise boundary delineation.
method Contextual Recursive Attention with Uncertainty Modeling, multi-scale context aggregation, attention mechanisms, edge-aware decoder, Monte Carlo Dropout.
result Superior performance in producing accurate and reliable saliency maps.
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
Enhances speech emotion recognition by integrating visual data with attention mechanisms.
problem Improving emotion detection accuracy by combining multiple modalities.
method Introducing an attention mechanism to combine audio, text, and video modalities.
result Significant improvement of 3.65% in weighted accuracy.
The paper presents a neural machine translation system for Wolof.
problem Low-resource conditions for translating French to Wolof.
method Developed a bidirectional LSTM-based encoder-decoder architecture with attention mechanisms.
result Best model achieved a 47% BLEU score for French-Wolof translation.
As an important part of speech recognition technology, automatic speech keyword recognition has been intensively studied in recent years. Such technology becomes especially pivotal under situations with limited infrastructures and computational resources, such as voice command recognition in vehicles and robot interact…
Unified method for local GBDT feature contributions.
problem Need for local model interpretation for GBDT.
method Unified computation mechanism for instance-level feature contributions.
result Unified computation mechanism for GBDT feature contributions.
Attention mechanisms and non-local mean operations in general are key ingredients in many state-of-the-art deep learning techniques. In particular, the Transformer model based on multi-head self-attention has recently achieved great success in natural language processing and computer vision. However, the vanilla algori…
The paper investigates how Global Self-attention improves GCNs.
problem Improving expressive power and addressing overfitting in GCNs.
method Applying Global Self-attention mechanism over graph features.
result GSA mechanism enhances expressive power and mitigates overfitting and over-smoothing in GCNs.
Graph Neural Networks (GNNs) are powerful to learn the representation of graph-structured data. Most of the GNNs use the message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information of its neighbors. To achieve a better expressive capability of node influences, attention m…
Graph attention is not always beneficial; conditions for perfect node classification are identified.
problem Understanding when graph attention mechanisms improve node classification performance.
method Theoretical analysis using Contextual Stochastic Block Models (CSBMs).
result Graph attention mechanisms are more effective when structure noise exceeds feature noise, and simpler graph convolution operations are better when feature noise predominates.
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domains to answer a certain question. Thus, robust attention mechanisms are essential for powerful VQA mo…
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, …
Attention mechanism is effective in both focusing the deep learning models on relevant features and interpreting them. However, attentions may be unreliable since the networks that generate them are often trained in a weakly-supervised manner. To overcome this limitation, we introduce the notion of input-dependent unce…