Compressive Transformer learns long-range sequences by compressing past memories.
problem Learning long-range sequences in language and speech models.
method Compressive Transformer compresses past memories for efficient long-range sequence learning.
result State-of-the-art performance on language and speech benchmarks.
LTM tackles long sequence language modeling by avoiding vanishing and exploding gradients.
problem Language models struggle with long sequences due to gradient issues.
method Introduces Long Term Memory network (LTM) that scales memory and weights input, avoiding overfitting.
result LTM achieves state-of-the-art perplexity results with fewer cells than previous models.
This paper improves RNN memory capacity for long sequences through learning associative memory update rules.
problem Challenges in RNNs remembering long sequences.
method Jointly learns memory update rule with task objective and uses multiple associative memories.
result Improves memory capacity for long sequence encoding.
Investigates long-range dependencies in language and music.
problem Measuring long-range dependence in sequence data.
method Statistical framework based on long memory stochastic processes.
result Testable implications about long memory in data and learned representations.
A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
Meta-learning improves event prediction from short sequences.
problem Predicting events from short sequences is challenging.
method Meta-learning approach using recurrent neural networks and monotonic neural networks.
result Meta-learning enhances long-term prediction performance.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
LEM efficiently models long-term sequences with gradients.
problem Learning long-term sequential dependencies in data.
method Gradient-based multiscale ordinary differential equations system.
result LEM outperforms state-of-the-art models in various tasks.
This paper improves RNNs' ability to handle long-term dependencies using an auxiliary loss.
problem Capturing long-term dependencies in RNNs is challenging.
method Adds an unsupervised auxiliary loss to improve long-term dependency capture.
result Improves performance and resource efficiency over competitive baselines.
A new method backtracks through a few key past states to speed up credit assignment in long sequences.
problem Computational inefficiency of back-propagation through time for long sequences.
method Sparse attentive backtracking using learned attention mechanisms to skip connections.
result Matches or outperforms regular BPTT and truncated BPTT in tasks with long-term dependencies.
Convolutional autoencoding improves long text reconstruction.
problem Text reconstruction quality decreases with text length.
method Sequence-to-sequence, purely convolutional and deconvolutional autoencoding.
result Better at reconstructing and correcting long paragraphs.
NAOMI improves imputation accuracy for long-range sequences.
problem Missing value imputation in spatiotemporal data.
method Non-autoregressive deep generative model exploiting multiresolution structure.
result Significant improvement in imputation accuracy (60% reduction in average prediction error).
New Performer model tackles long-sequence protein modeling.
problem Challenges of training complex Transformer models for long sequences.
method Linearly scalable long-context Transformer architecture, Performer.
result Performer provides strong theoretical guarantees and is effective for protein sequence modeling.
Method extracts knowledge from LSTM for sequence validation.
problem Validating sequences generated by unknown automata.
method Clustering hidden states to build a corresponding automaton.
result Automaton accurately predicts sequence validity.
Cloned HMMs efficiently learn variable order sequences without local minima issues.
problem Learning long-term temporal structure in sequences.
method Constrained HMMs with a sparsity structure that maps hidden states deterministically to emissions.
result Cloned HMMs can model temporal dependencies at arbitrarily long distances and recognize contexts with 'holes'.
Hrrformer uses HRR to speed up self-attention for long sequences.
problem Infeasibility of using transformers with very long sequence lengths.
method Re-cast self-attention using Holographic Reduced Representations (HRR).
result Achieves near state-of-the-art accuracy with O(THlogH) time complexity and O(TH) space complexity. A new LSTM model reduces state updates and improves convergence for long sequences.
problem Vanishing gradient problem in RNNs and slow convergence on long sequences.
method Gaussian-gated LSTM (g-LSTM) with a time gate to control neuron updates.
result The g-LSTM model reduces state updates and computes by at least 10x compared to an equivalent LSTM.
Novel graph neural network combines random walks with local message passing.
problem Graph neural networks struggle with long-range dependencies.
method Combines random walks with local message passing in a novel architecture.
result Significant performance improvements on graph benchmarks.
Several machine learning problems arising in natural language processing can be modeled as a sequence labeling problem. We provide Gaussian process models based on pseudo-likelihood approximation to perform sequence labeling. Gaussian processes (GPs) provide a Bayesian approach to learning in a kernel based framework. …
TRM improves long-horizon reinforcement learning for LLMs by masking divergent sequences.
problem Long-horizon reinforcement learning for LLMs suffers from off-policy mismatch and approximation errors.
method Derives and applies trust region bounds to control divergence, proposing Trust Region Masking.
result First non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.
Sparse Transformers reduce memory and time requirements for long sequence modeling.
problem Quadratic growth in memory and time with transformer sequence length.
method Sparse factorizations of attention matrix, deeper network training, attention matrix recomputation, fast attention kernels.
result Sparse Transformers can model sequences up to tens of thousands of timesteps with hundreds of layers.
New method improves deep learning model robustness and accuracy for long sequences.
problem Challenges in learning long-range sequence tasks using state-space models.
method Proposes a perturb-then-diagonalize (PTD) methodology to address ill-posed diagonalization problems in SSMs.
result Demonstrates improved robustness and accuracy of S5-PTD model on Long-Range Arena benchmark.
TRM improves long-horizon LLM RL by masking divergent sequences.
problem Long-horizon reinforcement learning with LLMs suffers from off-policy mismatch and approximation errors.
method Derives and applies trust region bounds to control divergence, proposing Trust Region Masking.
result First non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.
Spacetimeformer learns spatiotemporal relationships from data alone.
problem Forecasting multivariate time series with distinct spatial relationships.
method Transformers with dynamic graph connections learning interactions between space, time, and value.
result Competitive results on various time series prediction benchmarks.
RotRNN uses rotations to simplify long sequence modelling.
problem Complex initialisation and normalisation schemes in linear recurrent models.
method RotRNN employs rotation matrices to simplify and normalise linear recurrent models.
result RotRNN achieves competitive performance on long sequence modelling datasets.
TFiLM expands convolutional models' receptive field with minimal overhead.
problem Capturing long-range dependencies in sequential data.
method A novel architectural component using a recurrent neural network to modulate convolutional model activations.
result TFiLM significantly improves learning speed and accuracy on various tasks.
S3Attention improves long sequence attention with smoothed skeleton sketching.
problem Quadratic complexity of vanilla Attention makes it unsuitable for long sequence tasks.
method S3Attention uses smoothing and matrix sketching to balance information preservation and computation. result S3Attention significantly outperforms vanilla Attention and other Attention variants. A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
Neural M3 model adapts to diverse user behaviors over short and long timeframes.
problem Adapting to diverse user behaviors over short and long timeframes.
method Neural Multi-temporal-range Mixture Model (M3) combining short-term and long-term models with a learned gating mechanism.
result M3 consistently outperforms state-of-the-art sequential recommendation methods.
A new method predicts protein functions using variable-length sequences.
problem Computational methods for protein function prediction are slow and inaccurate for long sequences.
method Two feature sets: single fixed-sized segments and multi-sized segments, using bi-directional LSTM. Combined with MLDA features.
result Significant improvement in accuracy for long protein sequences.
New method maps protein sequences to embeddings encoding structural information.
problem Inferring structural properties from amino acid sequences when structures are unknown.
method Representation learning using bidirectional LSTM models with structural similarity and residue contact maps.
result Trained embeddings improve structural similarity prediction and transfer to other tasks.
Graph WaveNet models spatial-temporal graphs by learning hidden dependencies and long sequences.
problem Capturing hidden spatial dependencies and long-range temporal sequences in graphs.
method Graph WaveNet integrates adaptive dependency matrix learning and stacked dilated 1D convolution.
result Graph WaveNet outperforms existing methods on public traffic network datasets.
Neural Physicist learns physical dynamics from images.
problem Learning meaningful physical state representations and accurate state transitions from image sequences.
method Neural Physicist uses VAE for state extraction, NP for parameters, and SSM for dynamics.
result Achieves long-term predictions and identifies system degrees of freedom.
HGConv uses HRR to efficiently detect malware, outperforming existing methods.
problem Efficiently detecting malware with long sequences.
method Holographic Global Convolutional Networks (HGConv) utilizing Holographic Reduced Representations (HRR).
result Achieved state-of-the-art results on malware benchmarks.
Reformer improves Transformer efficiency for long sequences.
problem Prohibitively costly training of large Transformer models.
method Locality-sensitive hashing for attention and reversible residual layers.
result Memory-efficient and faster performance on long sequences.
Paper proposes using LSTM for LSH-based sequence alignment.
problem Sequence alignment using deep learning models.
method Deep bidirectional LSTM for feature learning and LSH-based sequence alignment.
result Higher accuracy achieved with LSTM-based model.
Improved GFlowNets learn more efficiently with trajectory balance.
problem Inefficient credit assignment in GFlowNets leads to suboptimal learning.
method Proposed trajectory balance as a new learning objective.
result Trajectory balance leads to more efficient and robust GFlowNet learning.
This paper studies mutual information in sequence models and finds Transformers excel in capturing long-range dependencies.
problem Understanding the expressive power of sequence models in capturing temporal dependencies.
method Theoretical and empirical analysis of linear and nonlinear RNNs, including Transformers.
result Transformers can capture long-range mutual information more efficiently than RNNs.
SSMs have a built-in bias towards low-frequency components, which can be adjusted.
problem Frequency bias in SSMs affects their performance on long-range sequences.
method Proposed two mechanisms to tune frequency bias: scaling initialization or applying a Sobolev-norm-based filter.
result Tuning frequency bias improves SSMs' performance on long-range sequence learning tasks.
Skeinformer accelerates self-attention for long sequences with linear complexity.
problem Efficiency of Transformer models in processing long sequences.
method Matrix sketching and column sampling to reduce quadratic complexity to linear.
result Skeinformer outperforms alternatives with smaller time/space footprint.
Improved text matching model using deconvolutional networks.
problem Text sequence matching challenges.
method Jointly optimizing generative and discriminative objectives with deconvolutional networks.
result Significantly outperforms sentence-encoding baselines, especially in semi-supervised settings.
Introduces Spectral Attention for better long-range time series forecasting.
problem Challenges in capturing long-range dependencies in time series forecasting.
method Spectral Attention mechanism that preserves temporal correlations and long-range dependencies.
result Achieves state-of-the-art results on 11 real-world time series datasets.
Enhanced LSTM model learns complex temporal dependencies.
problem Modeling long-term dependencies in sequential data.
method Gamma-LSTM with hierarchical memory units and gates.
result Gamma-LSTM outperforms regular and stacked LSTMs in sequence prediction.
MusicVAE uses a hierarchical decoder to model long-term structure in music sequences.
problem Difficulty of existing recurrent VAE models in modeling long-term structure in sequential data.
method Proposes a hierarchical decoder that outputs embeddings for subsequences and uses these embeddings to generate each subsequence independently.
result Demonstrates better sampling, interpolation, and reconstruction performance than a flat baseline model.
As high-throughput biological sequencing becomes faster and cheaper, the need to extract useful information from sequencing becomes ever more paramount, often limited by low-throughput experimental characterizations. For proteins, accurate prediction of their functions directly from their primary amino-acid sequences h…
IGLOO slices features space to process long sequences efficiently.
problem Efficiency in processing long sequences with neural networks.
method Slicing features space into non-local patches and using their relationships to build sequence representations.
result IGLOO can handle dependencies up to 20,000 steps efficiently.
VLSTM improves HFT by handling long sequences in financial trading.
problem Handling long sequences of thousands of data points in high-frequency trading.
method Proposed VLSTM, a variant of LSTM, to address long-term dependencies.
result 3.14\% increase in F1-score over state-of-the-art models.
D-LinOSS models learn to dissipate energy, improving performance on long-range tasks.
problem Representational limitations of LinOSS models in long-range reasoning.
method Introducing Damped Linear Oscillatory State-Space models (D-LinOSS) that learn to dissipate latent state energy on arbitrary time scales.
result D-LinOSS consistently outperforms previous LinOSS methods on long-range learning tasks, achieving faster convergence and reducing hyperparameter search space.