arXiv research

Clustered attention improves transformer efficiency for large sequences.

Attention operators have been widely applied in various fields, including computer vision, natural language processing, and network embedding learning. Attention operators on graph data enables learnable weights when aggregating information from neighboring nodes. However, graph attention operators (GAOs) consume exces…

Paper proposes an attention sampler for reducing attention mechanism computation.

S$^3$Attention improves long sequence attention with smoothed skeleton sketching.

Although deep neural networks generally have fixed network structures, the concept of dynamic mechanism has drawn more and more attention in recent years. Attention mechanisms compute input-dependent dynamic attention weights for aggregating a sequence of hidden states. Dynamic network configuration in convolutional ne…

Quaternion self-attention reduces computational cost and improves performance.

Study reveals attention mechanism's similarity computation parallels traditional machine learning.

Transformer models waste resources on long-context tasks.

New interpretation of attention in Transformers and Graph Attention Networks.

Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.

Skyformer uses Gaussian kernel and Nyström method to speed up self-attention in transformers.

Investigates the fundamental components of attention mechanisms.

Elliptical Attention improves transformer performance by focusing on contextually relevant features.

A brain-inspired spiking Transformer reduces energy consumption and enhances interpretability.

FastGAT reduces GNN computation time by 10x using graph sparsification.

This paper proposes an attention module augmented relational network called SARN(Sequential Attention Relational Network) that can carry out relational reasoning by extracting reference objects and making efficient pairing between objects. SARN greatly reduces the computational and memory requirements of the relational…

SpGAT learns graph representations using spectral attention for efficiency.

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…

This research integrates attention into XAI frameworks for better model explanations.

A new algorithm reduces memory usage for long token attention in streaming applications.

3D Axial-Attention improves lung nodule classification accuracy.

Efficient sparse attention reduces self-attention complexity and improves model performance.

GOAT improves attention mechanisms by learning better priors.

Hard visual attention is a promising approach to reduce the computational burden of modern computer vision methodologies. Hard attention mechanisms are typically non-differentiable. They can be trained with reinforcement learning but the high-variance training this entails hinders more widespread application. We show h…

ABRF uses attention weights to improve RF performance.

Centroid Transformers reduce memory and computation by summarizing inputs into centroids.

Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows contextual information to be exchanged among sequence elements. While many of the prev…

New algorithms speed up attention computation for large models by limiting matrix entries.

Deep neural network models have recently draw lots of attention, as it consistently produce impressive results in many computer vision tasks such as image classification, object detection, etc. However, interpreting such model and show the reason why it performs quite well becomes a challenging question. In this paper,…

Study integrates attentional and spacing factors to improve category learning models.

Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalit…

Attention mechanisms and non-local mean operations in general are key ingredients in many state-of-the-art deep learning techniques. In particular, the Transformer model based on multi-head self-attention has recently achieved great success in natural language processing and computer vision. However, the vanilla algori…

A framework for transformer attention layers derived from SVR.

Random forests with attention and self-attention improve regression performance.

Kernel PCA explains self-attention mechanisms in deep learning models.

LUNA improves linear attention for long sequences without sacrificing accuracy.

This work generalizes transformer attention to capture higher-order correlations efficiently.

In this paper, we propose Dynamic Self-Attention (DSA), a new self-attention mechanism for sentence embedding. We design DSA by modifying dynamic routing in capsule network (Sabouretal.,2017) for natural language processing. DSA attends to informative words with a dynamic weight vector. We achieve new state-of-the-art …

LARF improves random forests with attention mechanisms and contamination models.

LoLCATs improves linearized LLM quality with less memory and compute.

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attenti…

Improved Transformer performance by addressing 'explaining away' effect.

MAGNA improves graph neural networks by incorporating multi-hop context information.

A study on optimizing self-attention in tabular data using Optimal Transport.

Sparse Transformers can approximate dense Transformers with only O(n) connections.

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character lev…

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptiv…

Existing deep architectures cannot operate on very large signals such as megapixel images due to computational and memory constraints. To tackle this limitation, we propose a fully differentiable end-to-end trainable model that samples and processes only a fraction of the full resolution input image. The locations to p…