arXiv research

Introduces Spectral Attention for better long-range time series forecasting.

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also fi…

Novel graph neural network combines random walks with local message passing.

Missing value imputation is a fundamental problem in spatiotemporal modeling, from motion tracking to the dynamics of physical systems. Deep autoregressive models suffer from error propagation which becomes catastrophic for imputing long-range sequences. In this paper, we take a non-autoregressive approach and propose …

In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…

Improved Mamba model for long-range sequence tasks.

Rough Transformers improve efficiency for medical time-series data.

Representation and learning of long-range dependencies is a central challenge confronted in modern applications of machine learning to sequence data. Yet despite the prominence of this issue, the basic problem of measuring long-range dependence, either in a given data source or as represented in a trained deep model, r…

Long-range correlation and fluctuation in the gold market time series of world's two leading gold consuming countries, namely China and India, are studied. For both the market series during the period 1985-2013 we observe a long-range persistence of memory in the sequences of maxima (minima) of returns in successive ti…

Several machine learning problems arising in natural language processing can be modeled as a sequence labeling problem. We provide Gaussian process models based on pseudo-likelihood approximation to perform sequence labeling. Gaussian processes (GPs) provide a Bayesian approach to learning in a kernel based framework. …

HGConv uses HRR to efficiently detect malware, outperforming existing methods.

Self-attention prefers sparse functions of input sequences, reducing sample complexity.

SSMs have a built-in bias towards low-frequency components, which can be adjusted.

D-LinOSS models learn to dissipate energy, improving performance on long-range tasks.

Spacetimeformer learns spatiotemporal relationships from data alone.

S4 model improves long sequence modeling efficiency.

Cohesion uses deep Koopman operators to generate long-range forecasts of chaotic dynamics.

New method improves deep learning model robustness and accuracy for long sequences.

New model captures long-range patterns in sequences efficiently.

In this paper we present a new framework for time-series modeling that combines the best of traditional statistical models and neural networks. We focus on time-series with long-range dependencies, needed for monitoring fine granularity data (e.g. minutes, seconds, milliseconds), prevalent in operational use-cases. Tra…

Learning representations that accurately capture long-range dependencies in sequential inputs -- including text, audio, and genomic data -- is a key problem in deep learning. Feed-forward convolutional models capture only feature interactions within finite receptive fields while recurrent architectures can be slow and …

We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limit…

A method for estimating the cross-correlation $C_{xy}(τ)$ of long-range correlated series $x(t)$ and $y(t)$, at varying lags $τ$ and scales $n$, is proposed. For fractional Brownian motions with Hurst exponents $H_1$ and $H_2$, the asymptotic expression of $C_{xy}(τ)$ depends only on the lag $τ$ (wide-sense stationarit…

A method to generate long-range human actions by leveraging graph convolutional networks and self-attention.

MambaLRP enhances Mamba models' explainability and performance.

Understanding temporal dynamics has proved to be highly valuable for accurate recommendation. Sequential recommenders have been successful in modeling the dynamics of users and items over time. However, while different model architectures excel at capturing various temporal ranges or dynamics, distinct application cont…

Rough Transformers improve time series modeling with lower costs and better performance.

Autoregressive generative models consistently achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditio…

Convolutional Neural Networks (CNNs) have been recently introduced in the domain of session-based next item recommendation. An ordered collection of past items the user has interacted with in a session (or sequence) are embedded into a 2-dimensional latent matrix, and treated as an image. The convolution and pooling op…

Conditional random fields (CRFs) have been shown to be one of the most successful approaches to sequence labeling. Various linear-chain neural CRFs (NCRFs) are developed to implement the non-linear node potentials in CRFs, but still keeping the linear-chain hidden structure. In this paper, we propose NCRF transducers, …

We present the mixture-of-parents maximum entropy Markov model (MoP-MEMM), a class of directed graphical models extending MEMMs. The MoP-MEMM allows tractable incorporation of long-range dependencies between nodes by restricting the conditional distribution of each node to be a mixture of distributions given the parent…

Recently exciting progress has been made on protein contact prediction, but the predicted contacts for proteins without many sequence homologs is still of low quality and not very useful for de novo structure prediction. This paper presents a new deep learning method that predicts contacts by integrating both evolution…

The standard LSTM recurrent neural networks while very powerful in long-range dependency sequence applications have highly complex structure and relatively large (adaptive) parameters. In this work, we present empirical comparison between the standard LSTM recurrent neural network architecture and three new parameter-r…

Given the emerging global threat of antimicrobial resistance, new methods for next-generation antimicrobial design are urgently needed. We report a peptide generation framework PepCVAE, based on a semi-supervised variational autoencoder (VAE) model, for designing novel antimicrobial peptide (AMP) sequences. Our model l…

Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks …

Bayesian context trees capture complex dependencies in categorical sequences.

Characterizing temporal dependence patterns is a critical step in understanding the statistical properties of sequential data. Long Range Dependence (LRD) --- referring to long-range correlations decaying as a power law rather than exponentially w.r.t. distance --- demands a different set of tools for modeling the unde…

This paper looks into the analysis of the long-range auto-correlations and cross-correlations in bond market. Based on Detrended Moving Average (DMA) method, empirical results present a clear evidence of long-range persistence that exists in one year scale. The degree of long-range correlation related to maturities has…

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syn…

Skeinformer accelerates self-attention for long sequences with linear complexity.

Research on long-range memory in financial and social systems using various models.

Paper proposes using LSTM for LSH-based sequence alignment.

A new algorithm for restless bandits handles long-range dependencies.

A new method uses burst and inter-burst duration to test long-range memory in financial markets.

The paper provides a theoretical justification for using stable SSM blocks in deep sequential models.

Transformers can perform well with less long-range memory.

LUNA improves linear attention for long sequences without sacrificing accuracy.

Recurrent Neural Networks (RNNS) are now widely used on sequence generation tasks due to their ability to learn long-range dependencies and to generate sequences of arbitrary length. However, their left-to-right generation procedure only allows a limited control from a potential user which makes them unsuitable for int…