PBO methods improve RNN performance in learning long-term dependencies.
problem Training RNNs to learn long-term dependencies is challenging.
method Population-based global optimisation (PBO) techniques, including evolution strategies and particle swarm optimisation.
result PBO methods lead to performance improvements in RNNs for volatility forecasting.
Combines CNN and Transformer for financial time series forecasting.
problem Forecasting financial time series, especially stock prices, is challenging due to short-term and long-term dependencies.
method Uses CNN for short-term dependencies and Transformer for long-term dependencies.
result Demonstrated superior performance in forecasting stock price changes compared to traditional methods.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
TimeBridge addresses non-stationarity in long-term time series forecasting.
problem Non-stationarity in multivariate time series leads to spurious regressions and obscures long-term relationships.
method TimeBridge segments series into patches, applying Integrated Attention for short-term non-stationarity and Cointegrated Attention for long-term cointegration.
result TimeBridge achieves state-of-the-art performance in both short-term and long-term forecasting.
Sparse Attentive Backtracking selectively backpropagates long-term dependencies in recurrent networks.
problem Difficulty in learning long-term dependencies in BPTT due to computational impracticality and biased gradient estimates.
method Sparse Attentive Backtracking learns an attention mechanism over past hidden states and selectively backpropagates through high-weight paths.
result Model learns long-term dependencies with fewer backpropagation steps, addressing biased gradient issues.
GORU combines unitary and gated RNNs for better long-term memory management.
problem Learning to effectively manage long-term memory in neural networks.
method Extending unitary RNNs with a gating mechanism to forget irrelevant information.
result GORU outperforms LSTMs, GRUs, and Unitary RNNs on long-term dependency tasks.
Improved GRU model with weighted time-delay feedback for long-term dependencies.
problem Modeling long-term dependencies in sequential data.
method Introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism.
result τ-GRU outperforms state-of-the-art models on various tasks.
CFTM uses fractional Brownian motion for dynamic topic modeling.
problem Identifying long-term dependency or roughness in topic and word distributions over time.
method Continuous Time Fractional Topic Model (cFTM) incorporating fractional Brownian motion.
result cFTM captures long-term dependency or roughness in topic and word distributions.
New RNN model handles long-term dependencies in irregularly-sampled time series.
problem Handling long-term dependencies in irregularly-sampled time series data.
method Designing ODE-LSTMs that separate memory from continuous-time state.
result ODE-LSTMs outperform other RNN-based models on non-uniformly sampled data with long-term dependencies.
Paper introduces a simpler gated RNN structure to better capture long-term dependencies.
problem Difficulty in learning long-term dependencies in RNNs.
method Proposes a grouped distributor unit (GDU) with partitioned hidden states and adaptive update rates.
result GDU outperforms LSTM and GRU on various tasks, including pathological and natural data.
ST-SAN predicts flow with spatial-temporal dependencies using self-attention.
problem Challenges in predicting flow due to spatial-temporal dependencies.
method Spatial-Temporal Self-Attention Network (ST-SAN) that addresses temporal and spatial dependencies.
result Significant improvement in flow prediction accuracy (9% in inflow, 4% in outflow) compared to state-of-the-art methods.
AntisymmetricRNN improves RNN trainability without extra computation.
problem Difficulty in learning long-term dependencies in RNNs.
method Connecting RNNs to ordinary differential equations and proposing AntisymmetricRNN.
result AntisymmetricRNN captures long-term dependencies more predictably and efficiently.
KEDformer improves long-term time series forecasting with seasonal-trend decomposition.
problem Accurate long-term predictions in energy, finance, and meteorology.
method Knowledge extraction-driven framework integrating seasonal-trend decomposition.
result KEDformer enhances model's ability to capture short-term and long-term patterns.
This paper improves RNNs' ability to handle long-term dependencies using an auxiliary loss.
problem Capturing long-term dependencies in RNNs is challenging.
method Adds an unsupervised auxiliary loss to improve long-term dependency capture.
result Improves performance and resource efficiency over competitive baselines.
Proposes FRU to stabilize gradients and improve long-term dependencies in RNNs.
problem Challenges in training RNNs for tasks with long-term dependencies.
method Introduces Fourier Recurrent Units (FRU) that stabilize gradients and improve expressivity.
result FRU stabilizes gradients and has stronger expressive power, leading to better performance.
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
LEM efficiently models long-term sequences with gradients.
problem Learning long-term sequential dependencies in data.
method Gradient-based multiscale ordinary differential equations system.
result LEM outperforms state-of-the-art models in various tasks.
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
Memory augmented neural networks improve long-term dependency learning.
problem Vanishing gradients in RNNs for long-term dependency tasks.
method TARDIS model with wormhole connections to external memory.
result Memory helps propagate gradients effectively, improving learning.
New method improves learning of long-term dependencies in recurrent models.
problem Improving learning of long-term dependencies in recurrent neural networks.
method Proves learnable gates in recurrent models provide quasi-invariance to time transformations and recovers part of LSTM architecture from axiomatic approach.
result New chrono initialization of gate biases greatly improves learning of long-term dependencies.
SRU learns long-term dependencies without gates, outperforming LSTMs and GRUs.
problem Learning long-term dependencies in data efficiently.
method Developed an un-gated statistical recurrent unit (SRU) that keeps moving averages of statistics.
result SRU outperforms LSTMs and GRUs in various tasks, often outperforming both.
TiDE uses MLP for fast, simple long-term time-series forecasting.
problem Long-term time-series forecasting challenges.
method Time-series Dense Encoder (TiDE) based on MLP.
result TiDE matches or outperforms Transformer models while being 5-10x faster.
Paper proposes a graph neural network for accurate long-term ILI prediction.
problem Limited long-term prediction performance and spatio-temporal dependency in existing models.
method Cross-location attention based graph neural network (Cola-GNN) for time series embeddings and location aware attentions.
result Proposed method shows strong predictive performance and interpretable results for long-term epidemic predictions.
DSTP-RNN improves long-term multivariate time series prediction using attention-based RNN.
problem Long-term prediction of multivariate time series with spatial correlations and spatio-temporal relationships.
method Inspired by human attention mechanism, DSTP-RNN uses a dual-stage two-phase structure and multiple attentions to enhance spatial correlations and long-term dependence.
result DSTP-RNN outperforms nine baseline methods on four datasets in energy, finance, environment, and medicine.
Proposes RMN for learning long-term dependencies in feed-forward networks.
problem Complicated training of deep RNN architectures.
method Residual Memory Neural Network (RMN) with residual and time-delayed connections.
result RMN and BRMN outperform LSTM and BLSTM networks in learning long-term and hierarchical information.
Mamba Hawkes Process improves modeling of event sequences with long-term dependencies.
problem Modeling mutual inhibition and nonlinearity in asynchronous event sequences.
method Introduces Mamba Hawkes Process using Mamba state space architecture.
result MHP outperforms existing models across various datasets.
Transformer learns long-term dependencies from real-world data.
problem Sample inefficiency in deep reinforcement learning.
method Transformer architecture applied to autoregressive real-world episodes.
result Transformer-based world model generates meaningful experience.
Preformer improves Transformer for long-term time series forecasting.
problem Transformer's quadratic complexity and lack of context-awareness for long-term forecasting.
method Introduces Multi-Scale Segment-Correlation mechanism for efficient time series segmentation and context-aware attention.
result Preformer outperforms other Transformer-based methods in long-term time series forecasting.
KF-RTRL approximates RTRL for online learning of long-term dependencies.
problem Lack of efficient algorithms for learning long-term dependencies in RNNs.
method KF-RTRL uses Kronecker factorization to approximate RTRL gradients.
result KF-RTRL is an unbiased, memory-efficient online learning algorithm with lower noise than UORO.
Improved recurrent neural networks learn long-term dependencies through multi-scale memory.
problem Capturing long-term dependencies in recurrent neural networks.
method Incremental training of a modular RNN architecture with multi-scale hidden states.
result Incremental training and multi-scale memory enhance RNNs' ability to learn long-term dependencies.
Structured state space models improve ECG classification and reveal new insights.
problem Improving ECG analysis through deep learning.
method Applying structured state space models to capture long-term dependencies in ECG data.
result SSMs lead to significant improvements in ECG classification over current state-of-the-art.
Study finds long-term linear correlations in Chinese stock order aggressiveness.
problem Investigating long-term correlations in order aggressiveness of Chinese stocks.
method Used detrending moving average and multifractal detrending moving average analyses on order flow data.
result Strong long-term linear correlations found in order aggressiveness, with some exceptions.
THP model captures complex dependencies in event sequences efficiently.
problem Inability of existing models to capture long-term dependencies in event sequences.
method Transformer Hawkes Process (THP) model using self-attention mechanism.
result THP outperforms existing models in likelihood and event prediction accuracy.
Introduces LAMP for efficient modeling of long-term dependencies.
problem Efficiently modeling long-term dependencies in sequential data.
method Introduces Linear Additive Markov Process (LAMP) with efficient history dependence learning.
result LAMP models outperform first-order Markov processes and even deep sequential models.
FPG uses fractional calculus for efficient reinforcement learning with long-term memory.
problem High variance and inefficient sampling in standard policy gradient methods for long-term temporal modeling.
method Fractional Policy Gradients (FPG) incorporating Caputo fractional derivatives for power-law temporal correlations.
result Achieves asymptotic variance reduction of order O(t^(-alpha)) and sample efficiency gains.
Sequence feature embedding is a challenging task due to the unstructuredness of sequence, i.e., arbitrary strings of arbitrary length. Existing methods are efficient in extracting short-term dependencies but typically suffer from computation issues for the long-term. Sequence Graph Transform (SGT), a feature embedding …
Time-related features improve time series forecasting models.
problem Lack of explicit time-related encoding in current forecasting models limits their ability to capture cyclical and seasonal trends.
method Introducing Time Stamp Forecaster (TimeSter) to encode time-related features and integrating it with a linear backbone.
result TimeLinear model reduces MSE by 23% on benchmark datasets, improving performance with exceptional efficiency.
Hamiltonian RNN controls hidden states gradient for long-term dependencies.
problem Challenges in learning long-term dependencies in RNNs.
method Symplectic discretization of Hamiltonian system to control gradient.
result Hamiltonian RNN outperforms other RNNs without hyperparameter optimization.
A new memory-based fusion layer improves multi-modal deep learning performance.
problem Improving performance of multi-modal deep learning by addressing long-term dependencies.
method Introducing a Memory based Attentive Fusion (MBAF) layer that incorporates both current and long-term dependencies.
result The MBAF layer enhances fusion and improves performance across different modalities and networks.
Deep RNN model improves long-term BP prediction accuracy.
problem Long-term accuracy decay in BP prediction models.
method Formulated BP estimation as sequence prediction problem; used multilayered LSTM with bidirectional and residual connections.
result Deep RNN model outperformed traditional models in long-term BP prediction.
ERNNs evolve hidden states on an ODE's equilibrium manifold to mitigate vanishing and exploding gradients.
problem Vanishing and exploding gradients in RNNs.
method Develop a novel family of RNNs (ERNNs) that evolve hidden states on the equilibrium manifold of an ODE.
result ERNNs achieve state-of-the-art accuracy with 3-10x speedups and 1.5-3x model size reduction.
New algorithm optimizes for long-term user satisfaction in delayed reward settings.
problem Optimizing for long-term user satisfaction in delayed reward settings.
method Developed a predictive model of delayed rewards and a bandit algorithm that combines rewards and surrogate outcomes.
result Our algorithm significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards.
We propose a particularly structured Boltzmann machine, which we refer to as a dynamic Boltzmann machine (DyBM), as a stochastic model of a multi-dimensional time-series. The DyBM can have infinitely many layers of units but allows exact and efficient inference and learning when its parameters have a proposed structure…
We consider online optimization in the 1-lookahead setting, where the objective does not decompose additively over the rounds of the online game. The resulting formulation enables us to deal with non-stationary and/or long-term constraints , which arise, for example, in online display advertising problems. We propose a…
New interpretation of RNN forget gate improves learnability for long-term sequential data.
problem Improving learnability of recurrent neural networks for long-term temporal dependencies.
method Generalized theory of gated RNNs, focusing on gradient behavior over time.
result Existing RNNs satisfy the gradient condition for initial training, suggesting validity of forget gate interpretation.
Investment strategies differ based on short-term and long-term market time scales.
problem Identifying and understanding different time scales in stock market dynamics.
method Empirical Mode Decomposition (EMD) and Hurst Exponent analysis.
result Short-term market dynamics are random, while long-term are correlated with company fundamentals.
Improves sequence generation by training a backward network.
problem Generating long-term dependencies in sequence models.
method Train a backward recurrent network to predict states of a forward model.
result Achieves 9% relative improvement in speech recognition and significant improvement in caption generation.
The study improves Monte Carlo simulations for long-term investments using advanced financial models.
problem Improving the accuracy of long-term investment simulations.
method Developed a multivariate process incorporating recent financial models and probabilistic forecasts.
result Increased accuracy in predicting portfolio values over decades.