arXiv research

Proposes a model combining difference-attention and error-correction LSTMs for improved time series prediction.

Improved speech enhancement with MNTFA using time-frequency attention.

GACAN combines multi-granularity time series for traffic forecasting.

ANCDEs improve time-series forecasting and classification using attention in NCDEs.

Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalit…

New approach improves linear-time attention for language models.

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptiv…

In (Yang et al. 2016), a hierarchical attention network (HAN) is created for document classification. The attention layer can be used to visualize text influential in classifying the document, thereby explaining the model's prediction. We successfully applied HAN to a sequential analysis task in the form of real-time m…

WAVE improves time series forecasting by integrating AR and MA components.

Generative adversarial networks with attention improve financial time series simulation.

Transformers with linear space and time complexity for accurate attention estimation.

A new LSTM architecture improves time series forecasting efficiency.

Linformer reduces transformer complexity to linear, improving efficiency.

New method uses shared attention for multi-task time series forecasting.

Introduces Spectral Attention for better long-range time series forecasting.

Financial time series prediction, especially with machine learning techniques, is an extensive field of study. In recent times, deep learning methods (especially time series analysis) have performed outstandingly for various industrial problems, with better prediction than machine learning methods. Moreover, many resea…

Time series prediction with deep learning methods, especially long short-term memory neural networks (LSTMs), have scored significant achievements in recent years. Despite the fact that the LSTMs can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within mu…

Sequential modelling with self-attention has achieved cutting edge performances in natural language processing. With advantages in model flexibility, computation complexity and interpretability, self-attention is gradually becoming a key component in event sequence models. However, like most other sequence models, self…

Conformer encoder reverses sequence in time dimension, affecting decoder training.

Paper proposes a forecasting model combining autoregressive models with spectral attention.

Introduces recency bias to improve time-series forecasting.

While LSTMs show increasingly promising results for forecasting Financial Time Series (FTS), this paper seeks to assess if attention mechanisms can further improve performance. The hypothesis is that attention can help prevent long-term dependencies experienced by LSTM models. To test this hypothesis, the main contribu…

Topological attention improves forecasting of univariate time series.

TaLK Convolutions improve sequence modeling efficiency.

Linear attention in Transformers can be interpreted as dynamic VAR models.

Unified framework for sequence models using test-time regression.

Intelligent Momentum Transformer outperforms traditional trading strategies.

New algorithms speed up attention computation for large models by limiting matrix entries.

A new algorithm reduces memory usage for long token attention in streaming applications.

A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason,…

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…

Forecasting multivariate time series data, such as prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate the task. To obtain accurate prediction, it is cru…

TACTiS models time series uncertainty with transformer attention.

A new method calibrates scientific models by adding randomness to their predictions.

Structured State-Space Duality connects SSMs to masked attention.

Neural nets predict user attention from mouse movements.

New model recognizes emotions with missing modalities, improving accuracy.

KATA improves associative recall by optimizing feature maps derived from nonnegative attention weights.

Quantum self-attention boosts automated market maker performance in crypto trading.

Recent efforts to improve the interpretability of deep neural networks use saliency to characterize the importance of input features to predictions made by models. Work on interpretability using saliency-based methods on Recurrent Neural Networks (RNNs) has mostly targeted language tasks, and their applicability to tim…

STAM learns important time steps and variables for multivariate time series prediction.

Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages or communication errors. …

Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows contextual information to be exchanged among sequence elements. While many of the prev…

Paper proposes an attention sampler for reducing attention mechanism computation.

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them ca…

New approach improves multi-head attention by making heads less similar.

FMA-ETA predicts travel time using FFN with attention.

Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is to capture the spatial correlations at the same time, the spatio-temporal relationships at different times and the long-term dependence of the temporal relationships between different series.…