LTM tackles long sequence language modeling by avoiding vanishing and exploding gradients.
problem Language models struggle with long sequences due to gradient issues.
method Introduces Long Term Memory network (LTM) that scales memory and weights input, avoiding overfitting.
result LTM achieves state-of-the-art perplexity results with fewer cells than previous models.
Compressive Transformer learns long-range sequences by compressing past memories.
problem Learning long-range sequences in language and speech models.
method Compressive Transformer compresses past memories for efficient long-range sequence learning.
result State-of-the-art performance on language and speech benchmarks.
RotRNN uses rotations to simplify long sequence modelling.
problem Complex initialisation and normalisation schemes in linear recurrent models.
method RotRNN employs rotation matrices to simplify and normalise linear recurrent models.
result RotRNN achieves competitive performance on long sequence modelling datasets.
S3Attention improves long sequence attention with smoothed skeleton sketching.
problem Quadratic complexity of vanilla Attention makes it unsuitable for long sequence tasks.
method S3Attention uses smoothing and matrix sketching to balance information preservation and computation. result S3Attention significantly outperforms vanilla Attention and other Attention variants. A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
Investigates long-range dependencies in language and music.
problem Measuring long-range dependence in sequence data.
method Statistical framework based on long memory stochastic processes.
result Testable implications about long memory in data and learned representations.
Reformer improves Transformer efficiency for long sequences.
problem Prohibitively costly training of large Transformer models.
method Locality-sensitive hashing for attention and reversible residual layers.
result Memory-efficient and faster performance on long sequences.
NAOMI improves imputation accuracy for long-range sequences.
problem Missing value imputation in spatiotemporal data.
method Non-autoregressive deep generative model exploiting multiresolution structure.
result Significant improvement in imputation accuracy (60% reduction in average prediction error).
This paper studies mutual information in sequence models and finds Transformers excel in capturing long-range dependencies.
problem Understanding the expressive power of sequence models in capturing temporal dependencies.
method Theoretical and empirical analysis of linear and nonlinear RNNs, including Transformers.
result Transformers can capture long-range mutual information more efficiently than RNNs.
Skeinformer accelerates self-attention for long sequences with linear complexity.
problem Efficiency of Transformer models in processing long sequences.
method Matrix sketching and column sampling to reduce quadratic complexity to linear.
result Skeinformer outperforms alternatives with smaller time/space footprint.
Introduces Spectral Attention for better long-range time series forecasting.
problem Challenges in capturing long-range dependencies in time series forecasting.
method Spectral Attention mechanism that preserves temporal correlations and long-range dependencies.
result Achieves state-of-the-art results on 11 real-world time series datasets.
Novel graph neural network combines random walks with local message passing.
problem Graph neural networks struggle with long-range dependencies.
method Combines random walks with local message passing in a novel architecture.
result Significant performance improvements on graph benchmarks.
Meta-learning improves event prediction from short sequences.
problem Predicting events from short sequences is challenging.
method Meta-learning approach using recurrent neural networks and monotonic neural networks.
result Meta-learning enhances long-term prediction performance.
Method extracts knowledge from LSTM for sequence validation.
problem Validating sequences generated by unknown automata.
method Clustering hidden states to build a corresponding automaton.
result Automaton accurately predicts sequence validity.
A new method backtracks through a few key past states to speed up credit assignment in long sequences.
problem Computational inefficiency of back-propagation through time for long sequences.
method Sparse attentive backtracking using learned attention mechanisms to skip connections.
result Matches or outperforms regular BPTT and truncated BPTT in tasks with long-term dependencies.
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability …
New Performer model tackles long-sequence protein modeling.
problem Challenges of training complex Transformer models for long sequences.
method Linearly scalable long-context Transformer architecture, Performer.
result Performer provides strong theoretical guarantees and is effective for protein sequence modeling.
LEM efficiently models long-term sequences with gradients.
problem Learning long-term sequential dependencies in data.
method Gradient-based multiscale ordinary differential equations system.
result LEM outperforms state-of-the-art models in various tasks.
Sequence feature embedding is a challenging task due to the unstructuredness of sequence, i.e., arbitrary strings of arbitrary length. Existing methods are efficient in extracting short-term dependencies but typically suffer from computation issues for the long-term. Sequence Graph Transform (SGT), a feature embedding …
Hrrformer uses HRR to speed up self-attention for long sequences.
problem Infeasibility of using transformers with very long sequence lengths.
method Re-cast self-attention using Holographic Reduced Representations (HRR).
result Achieves near state-of-the-art accuracy with O(THlogH) time complexity and O(TH) space complexity. Sparse Transformers reduce memory and time requirements for long sequence modeling.
problem Quadratic growth in memory and time with transformer sequence length.
method Sparse factorizations of attention matrix, deeper network training, attention matrix recomputation, fast attention kernels.
result Sparse Transformers can model sequences up to tens of thousands of timesteps with hundreds of layers.
A new LSTM model reduces state updates and improves convergence for long sequences.
problem Vanishing gradient problem in RNNs and slow convergence on long sequences.
method Gaussian-gated LSTM (g-LSTM) with a time gate to control neuron updates.
result The g-LSTM model reduces state updates and computes by at least 10x compared to an equivalent LSTM.
Develops a hybrid model for text summarization.
problem Summarizing long text sequences concisely.
method Extends sequence encoders with a graph component to handle long-distance relationships in text.
result Hybrid models outperform pure sequence or graph models on summarization tasks.
A new method predicts protein functions using variable-length sequences.
problem Computational methods for protein function prediction are slow and inaccurate for long sequences.
method Two feature sets: single fixed-sized segments and multi-sized segments, using bi-directional LSTM. Combined with MLDA features.
result Significant improvement in accuracy for long protein sequences.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
S4 model improves long sequence modeling efficiency.
problem Handling long-range dependencies efficiently in convolutional models.
method S4 model uses global convolution with decaying kernel weights.
result SGConv model achieves similar performance to S4 with improved efficiency.
Neural M3 model adapts to diverse user behaviors over short and long timeframes.
problem Adapting to diverse user behaviors over short and long timeframes.
method Neural Multi-temporal-range Mixture Model (M3) combining short-term and long-term models with a learned gating mechanism.
result M3 consistently outperforms state-of-the-art sequential recommendation methods.
We give a definition of symplectic homology for pairs of filled Liouville cobordisms, and show that it satisfies analogues of the Eilenberg-Steenrod axioms except for the dimension axiom. The resulting long exact sequence of a pair generalizes various earlier long exact sequences such as the handle attaching sequence, …
Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.
problem Efficiently scaling Transformer architecture to long sequences with limited compute resources.
method Combines shallow Transformer models with recurrent neural networks and persistent memory.
result CRT achieves comparable or superior performance to full-length Transformers with shorter segments and reduced FLOPs.
Cloned HMMs efficiently learn variable order sequences without local minima issues.
problem Learning long-term temporal structure in sequences.
method Constrained HMMs with a sparsity structure that maps hidden states deterministically to emissions.
result Cloned HMMs can model temporal dependencies at arbitrarily long distances and recognize contexts with 'holes'.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
Transformer model generates long musical compositions with compelling structure.
problem Creating long musical compositions with coherent structure using Transformer models.
method Modified relative attention mechanism to reduce memory complexity and enable generation of minute-long compositions.
result Transformer with modified relative attention generates minute-long compositions with compelling structure.
Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…
Learning latent representations from long text sequences is an important first step in many natural language processing applications. Recurrent Neural Networks (RNNs) have become a cornerstone for this challenging task. However, the quality of sentences during RNN-based decoding (reconstruction) decreases with the leng…
Holonomy defined for singular leaves in foliations.
problem Defining holonomy for singular structures in foliations.
method Introducing a sequence of group morphisms from π_n(L) to π_{n-1} of the universal Lie ∞-algebroid.
result Holonomy sequence relates to Androulidakis-Skandalis and Brahic-Zhu constructions.
TRM improves long-horizon LLM RL by masking divergent sequences.
problem Long-horizon reinforcement learning with LLMs suffers from off-policy mismatch and approximation errors.
method Derives and applies trust region bounds to control divergence, proposing Trust Region Masking.
result First non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.
TRM improves long-horizon reinforcement learning for LLMs by masking divergent sequences.
problem Long-horizon reinforcement learning for LLMs suffers from off-policy mismatch and approximation errors.
method Derives and applies trust region bounds to control divergence, proposing Trust Region Masking.
result First non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.
Several machine learning problems arising in natural language processing can be modeled as a sequence labeling problem. We provide Gaussian process models based on pseudo-likelihood approximation to perform sequence labeling. Gaussian processes (GPs) provide a Bayesian approach to learning in a kernel based framework. …
NCRF transducers improve sequence labeling across tasks.
problem Sequence labeling with non-linear node potentials.
method Two RNNs: one for features and one for long-range label dependencies.
result Consistent improvements over linear-chain NCRFs and RNN transducers.
VLSTM improves HFT by handling long sequences in financial trading.
problem Handling long sequences of thousands of data points in high-frequency trading.
method Proposed VLSTM, a variant of LSTM, to address long-term dependencies.
result 3.14\% increase in F1-score over state-of-the-art models.
ADD-THIN improves TPP forecasting by handling long-term data sequences.
problem Sequential limitations in autoregressive models for TPPs.
method Diffusion model for TPPs that operates on entire sequences.
result ADD-THIN outperforms state-of-the-art models in forecasting.
AverageTime uses simple averaging to enhance long-term time series forecasting.
problem Long-term time series forecasting with improved intra-sequence and cross-channel dependencies.
method Proposes AverageTime, a simple, efficient, and scalable forecasting model that reframes channel extraction as a stackable architecture.
result AverageTime surpasses state-of-the-art models in forecasting performance with near-linear complexity.
New method improves deep learning model robustness and accuracy for long sequences.
problem Challenges in learning long-range sequence tasks using state-space models.
method Proposes a perturb-then-diagonalize (PTD) methodology to address ill-posed diagonalization problems in SSMs.
result Demonstrates improved robustness and accuracy of S5-PTD model on Long-Range Arena benchmark.
In this paper we construct Cech cohomology groups that form a Gysin-type long exact sequence for principal torus bundles. This sequence is modeled on a de Rham cohomology sequence published in earlier work by Bouwknegt, Hannabuss and Mathai, which was developed to compute the global properties of T-duality in the prese…
Paper proposes using LSTM for LSH-based sequence alignment.
problem Sequence alignment using deep learning models.
method Deep bidirectional LSTM for feature learning and LSH-based sequence alignment.
result Higher accuracy achieved with LSTM-based model.
Paper shows how LSTM can remember long sequences by attending to persisted information.
problem LSTMs struggle with long sequences due to fading information and bias towards recent data.
method The paper introduces a mechanism that allows LSTMs to attend to information in memory based on how long it was persisted by the gating mechanism.
result The method improves LSTM's ability to process long sequences by retrieving information proportionally to its persistence in memory.
Rough Transformers improve efficiency for medical time-series data.
problem Efficiently modeling irregularly sampled, long-range time-series data.
method Introducing Rough Transformers, a Transformer variant with continuous-time representations and multi-view signature attention.
result Rough Transformers outperform vanilla Transformers while using less computational resources.
Sumformer simplifies Transformers to handle long sequences efficiently.
problem Quadratic complexity of Transformers limits their use with long sequences.
method Introducing Sumformer, a simple architecture that universally approximates equivariant sequence-to-sequence functions.
result Sumformer achieves the first universal approximation results for Linformer and Performer.