arXiv research

RotRNN uses rotations to simplify long sequence modelling.

Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also fi…

S$^3$Attention improves long sequence attention with smoothed skeleton sketching.

Introduces Spectral Attention for better long-range time series forecasting.

Skeinformer accelerates self-attention for long sequences with linear complexity.

New Performer model tackles long-sequence protein modeling.

Novel graph neural network combines random walks with local message passing.

LEM efficiently models long-term sequences with gradients.

S4 model improves long sequence modeling efficiency.

Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph compone…

Meta-learning improves event prediction from short sequences.

Missing value imputation is a fundamental problem in spatiotemporal modeling, from motion tracking to the dynamics of physical systems. Deep autoregressive models suffer from error propagation which becomes catastrophic for imputing long-range sequences. In this paper, we take a non-autoregressive approach and propose …

Representation and learning of long-range dependencies is a central challenge confronted in modern applications of machine learning to sequence data. Yet despite the prominence of this issue, the basic problem of measuring long-range dependence, either in a given data source or as represented in a trained deep model, r…

Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.

Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability …

Several machine learning problems arising in natural language processing can be modeled as a sequence labeling problem. We provide Gaussian process models based on pseudo-likelihood approximation to perform sequence labeling. Gaussian processes (GPs) provide a Bayesian approach to learning in a kernel based framework. …

In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…

Hrrformer uses HRR to speed up self-attention for long sequences.

ADD-THIN improves TPP forecasting by handling long-term data sequences.

New method improves deep learning model robustness and accuracy for long sequences.

Paper proposes using LSTM for LSH-based sequence alignment.

Rough Transformers improve efficiency for medical time-series data.

AverageTime uses simple averaging to enhance long-term time series forecasting.

TRM improves long-horizon LLM RL by masking divergent sequences.

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper net…

TRM improves long-horizon reinforcement learning for LLMs by masking divergent sequences.

Mamba Hawkes Process improves modeling of event sequences with long-term dependencies.

Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks …

Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.

Understanding temporal dynamics has proved to be highly valuable for accurate recommendation. Sequential recommenders have been successful in modeling the dynamics of users and items over time. However, while different model architectures excel at capturing various temporal ranges or dynamics, distinct application cont…

Variable order sequence modeling is an important problem in artificial and natural intelligence. While overcomplete Hidden Markov Models (HMMs), in theory, have the capacity to represent long-term temporal structure, they often fail to learn and converge to local minima. We show that by constraining HMMs with a simple …

In this paper we construct Cech cohomology groups that form a Gysin-type long exact sequence for principal torus bundles. This sequence is modeled on a de Rham cohomology sequence published in earlier work by Bouwknegt, Hannabuss and Mathai, which was developed to compute the global properties of T-duality in the prese…

Amino acid sequence portrays most intrinsic form of a protein and expresses primary structure of protein. The order of amino acids in a sequence enables a protein to acquire a particular stable conformation that is responsible for the functions of the protein. This relationship between a sequence and its function motiv…

MambaLRP enhances Mamba models' explainability and performance.

Study compares LSTM, Transformer, and Mamba for bladder cancer recurrence analysis.

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensiti…

Sequence feature embedding is a challenging task due to the unstructuredness of sequence, i.e., arbitrary strings of arbitrary length. Existing methods are efficient in extracting short-term dependencies but typically suffer from computation issues for the long-term. Sequence Graph Transform (SGT), a feature embedding …

KEDformer improves long-term time series forecasting with seasonal-trend decomposition.

Global Memory Augmentation (GMAT) improves Transformer performance on long documents.

Improved Mamba model for long-range sequence tasks.

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Me…

New model captures long-range patterns in sequences efficiently.

Spacetimeformer learns spatiotemporal relationships from data alone.

Goodwillie's model connects knot spaces to cosimplicial spaces, aiding in knot homotopy computation.

SSMs have a built-in bias towards low-frequency components, which can be adjusted.

D-LinOSS models learn to dissipate energy, improving performance on long-range tasks.

Factorial Hidden Markov Models (FHMMs) are powerful models for sequential data but they do not scale well with long sequences. We propose a scalable inference and learning algorithm for FHMMs that draws on ideas from the stochastic variational inference, neural network and copula literatures. Unlike existing approaches…