arXiv research

Proposes a novel SAM operator for separate item and relational memories.

Active-memory mechanisms can replace self-attention in Transformers, but optimal results often require both.

Transformers can store facts efficiently using associative memories.

LLMs can be tricked into recalling facts based on context clues.

Linformer reduces transformer complexity to linear, improving efficiency.

Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of…

Derives an empirical capacity model for self-attention neural networks.

Efficient sparse attention reduces self-attention complexity and improves model performance.

Improved speech enhancement with MNTFA using time-frequency attention.

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, …

Transformers predict price movements from limit order books.

OSA overcomes instability in skipless Transformers.

Centroid Transformers reduce memory and computation by summarizing inputs into centroids.

Hrrformer uses HRR to speed up self-attention for long sequences.

The paper uses GRU and self-attention for SPY option pricing.

V-HMN integrates memory mechanisms for improved image recognition.

Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.

Learning long-term dependencies in extended temporal sequences requires credit assignment to events far back in the past. The most common method for training recurrent neural networks, back-propagation through time (BPTT), requires credit information to be propagated backwards through every single step of the forward c…

A new perspective on self-attention models using MLPs.

Sequential modelling with self-attention has achieved cutting edge performances in natural language processing. With advantages in model flexibility, computation complexity and interpretability, self-attention is gradually becoming a key component in event sequence models. However, like most other sequence models, self…

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character lev…

Improved Transformer performance by addressing 'explaining away' effect.

Transformers become faster by linearizing self-attention.

Extended LSTM improves language modeling performance.

Method learns audio embeddings with contextualized tags.

Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling …

Study reveals AI's spontaneous topic changes in text prediction.

Study examines asset pricing using various attention models, finding global self-attention and sliding window sparse attention models perform well.

This paper offers an overview of neural network compression techniques.

DAM with MRL improves relational reasoning in MANNs.

Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.

GSA-Nets apply group equivariance to self-attention for vision tasks.

AttViz offers online visualizations of neural language model attention mechanisms.

Knowledge tracing is the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities. Each student's knowledge is modeled by estimating the performance of the student on the learning activities. It is an important research area for providing a personalized…

The paper studies scaling laws for associative memory mechanisms.

Paper proposes multiscale self-attentive convolutions for vision and language.

Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…

Self-attention prefers sparse functions of input sequences, reducing sample complexity.

Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.

Study algebraic invariants from lightning self-attention models.

Random forests with attention and self-attention improve regression performance.

Paper investigates Lipschitz constants of self-attention modules in neural networks.

This paper introduces a hierarchical associative memory model with multiple layers.

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper net…

Kernel PCA explains self-attention mechanisms in deep learning models.

Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalit…

Proposes a faster Transformer decoding method by truncating target-side self-attention windows.