arXiv research

Proposes a model combining difference-attention and error-correction LSTMs for improved time series prediction.

Improved speech enhancement with MNTFA using time-frequency attention.

GACAN combines multi-granularity time series for traffic forecasting.

ANCDEs improve time-series forecasting and classification using attention in NCDEs.

Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalit…

New method embeds time span into self-attention for better temporal pattern recognition.

New approach improves linear-time attention for language models.

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptiv…

In (Yang et al. 2016), a hierarchical attention network (HAN) is created for document classification. The attention layer can be used to visualize text influential in classifying the document, thereby explaining the model's prediction. We successfully applied HAN to a sequential analysis task in the form of real-time m…

WAVE improves time series forecasting by integrating AR and MA components.

Generative adversarial networks with attention improve financial time series simulation.

Transformers with linear space and time complexity for accurate attention estimation.

A new LSTM architecture improves time series forecasting efficiency.

Linformer reduces transformer complexity to linear, improving efficiency.

New method uses shared attention for multi-task time series forecasting.

Introduces Spectral Attention for better long-range time series forecasting.

Input-cell attention improves RNN saliency detection over time.

Financial time series prediction, especially with machine learning techniques, is an extensive field of study. In recent times, deep learning methods (especially time series analysis) have performed outstandingly for various industrial problems, with better prediction than machine learning methods. Moreover, many resea…

Time series prediction with deep learning methods, especially long short-term memory neural networks (LSTMs), have scored significant achievements in recent years. Despite the fact that the LSTMs can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within mu…

Conformer encoder reverses sequence in time dimension, affecting decoder training.

Paper proposes a forecasting model combining autoregressive models with spectral attention.

Introduces recency bias to improve time-series forecasting.

While LSTMs show increasingly promising results for forecasting Financial Time Series (FTS), this paper seeks to assess if attention mechanisms can further improve performance. The hypothesis is that attention can help prevent long-term dependencies experienced by LSTM models. To test this hypothesis, the main contribu…

Topological attention improves forecasting of univariate time series.

TaLK Convolutions improve sequence modeling efficiency.

Linear attention in Transformers can be interpreted as dynamic VAR models.

Unified framework for sequence models using test-time regression.

Intelligent Momentum Transformer outperforms traditional trading strategies.

New algorithms speed up attention computation for large models by limiting matrix entries.

A new algorithm reduces memory usage for long token attention in streaming applications.

A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason,…

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…

Forecasting multivariate time series data, such as prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate the task. To obtain accurate prediction, it is cru…

TACTiS models time series uncertainty with transformer attention.

A new method calibrates scientific models by adding randomness to their predictions.

Structured State-Space Duality connects SSMs to masked attention.

Neural nets predict user attention from mouse movements.

New model recognizes emotions with missing modalities, improving accuracy.

KATA improves associative recall by optimizing feature maps derived from nonnegative attention weights.

Quantum self-attention boosts automated market maker performance in crypto trading.

STAM learns important time steps and variables for multivariate time series prediction.

Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages or communication errors. …

Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows contextual information to be exchanged among sequence elements. While many of the prev…

Proposes a time-aware attention model for CTR prediction.

Paper proposes an attention sampler for reducing attention mechanism computation.

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them ca…

New approach improves multi-head attention by making heads less similar.

FMA-ETA predicts travel time using FFN with attention.