Model combines long-term and short-term memory using conceptors.
problem Transfer between long-term and short-term memory.
method Recurrent neural network with gated reservoir for short-term memory and conceptors for long-term memory.
result Standard operations on conceptors allow combining long-term memories and describing their effect on short-term memory.
We study the long-term memory in diverse stock market indices and foreign exchange rates using the Detrended Fluctuation Analysis(DFA). For all daily and high-frequency market data studied, no significant long-term memory property is detected in the return series, while a strong long-term memory property is found in th…
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
LEMN improves memory networks for learning from streaming data.
problem Efficiently processing large data streams in memory-augmented neural networks.
method LEMN uses a RNN-based retention agent to learn and replace less important memory entries based on their importance and historical context.
result LEMN achieves significant improvements over existing methods in learning from streaming data.
We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in t…
Stable Hadamard Memory improves reinforcement learning by efficiently managing memory.
problem Memory models struggle in partially observable reinforcement learning environments.
method Introduces a novel memory model using the Hadamard product for efficient memory management and updates.
result Significantly outperforms state-of-the-art memory-based methods on challenging benchmarks.
New algorithms provide long-term memory guarantees for online learning.
problem Achieving long-term memory in online learning for experts and bandits.
method Reduction to typical switching regret, developing various algorithms with specific regret bounds.
result Developed algorithms with a new regret bound of O ( T ( S ln T + n ln K ) ) \mathcal{O}(\sqrt{T(S\ln T + n \ln K)}) O ( T ( S ln T + n ln K ) ) . FPG uses fractional calculus for efficient reinforcement learning with long-term memory.
problem High variance and inefficient sampling in standard policy gradient methods for long-term temporal modeling.
method Fractional Policy Gradients (FPG) incorporating Caputo fractional derivatives for power-law temporal correlations.
result Achieves asymptotic variance reduction of order O(t^(-alpha)) and sample efficiency gains.
New algorithms for multitask learning with long-term memory.
problem Learning from tasks partitioned into unknown segments with associated hypotheses.
method Online multitask learning algorithms exploiting segmentation and hypothesis association.
result Regret bounds and efficient algorithms for various hypothesis classes.
A novel online GP model captures long-term memory in sequential data.
problem Capturing long-term memory in sequential data online.
method Integrates HiPPO framework into interdomain GP, leveraging time-varying orthogonal projections as inducing variables.
result OHSVGP outperforms existing online GP methods in predictive performance, long-term memory preservation, and computational efficiency.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
LEM efficiently models long-term sequences with gradients.
problem Learning long-term sequential dependencies in data.
method Gradient-based multiscale ordinary differential equations system.
result LEM outperforms state-of-the-art models in various tasks.
The paper introduces a new stochastic volatility model with long-term memory and jumps.
problem Developing a model for variance and volatility swaps with long-term memory and jumps.
method Fractional Barndorff-Nielsen and Shephard model incorporating long-term memory and jumps.
result Arbitrage-free prices for variance and volatility swaps derived for the new model.
LTM tackles long sequence language modeling by avoiding vanishing and exploding gradients.
problem Language models struggle with long sequences due to gradient issues.
method Introduces Long Term Memory network (LTM) that scales memory and weights input, avoiding overfitting.
result LTM achieves state-of-the-art perplexity results with fewer cells than previous models.
Efficient algorithm predicts unknown linear systems with long-term memory.
problem Predicting unknown and partially observed linear dynamical systems with long-term memory.
method Bounding the generalized Kolmogorov width of the Kalman filter model using spectral methods and conducting tight convex relaxation.
result Competes with Kalman filter in hindsight with only logarithmic regret.
We focus on emergence of the power-law cross-correlations from processes with both short and long term memory properties. In the case of correlated error-terms, the power-law decay of the cross-correlation function comes automatically with the characteristics of separate processes. Bivariate Hurst exponent is then equa…
fSDE-Net generates time series with long-term memory using neural networks.
problem Generating time series with long-term memory from irregularly sampled data.
method fSDE-Net: neural fractional Stochastic Differential Equation Network using fractional Brownian motion.
result fSDE-Net can replicate distributional properties of real time-series data.
New RNN model handles long-term dependencies in irregularly-sampled time series.
problem Handling long-term dependencies in irregularly-sampled time series data.
method Designing ODE-LSTMs that separate memory from continuous-time state.
result ODE-LSTMs outperform other RNN-based models on non-uniformly sampled data with long-term dependencies.
In this paper we outline initial concepts for an immune inspired algorithm to evaluate price time series data. The proposed solution evolves a short term pool of trackers dynamically through a process of proliferation and mutation, with each member attempting to map to trends in price movements. Successful trackers fee…
Recent empirical results on long-term dependency tasks have shown that neural networks augmented with an external memory can learn the long-term dependency tasks more easily and achieve better generalization than vanilla recurrent neural networks (RNN). We suggest that memory augmented neural networks can reduce the ef…
Improved recurrent neural networks learn long-term dependencies through multi-scale memory.
problem Capturing long-term dependencies in recurrent neural networks.
method Incremental training of a modular RNN architecture with multi-scale hidden states.
result Incremental training and multi-scale memory enhance RNNs' ability to learn long-term dependencies.
Language models are miscalibrated, leading to overestimated entropy rates and memory usage.
problem Miscalibration of language models leading to overestimated entropy rates and memory usage.
method Calibration-based approach to measure discrepancies and improve models.
result State-of-the-art language models are miscalibrated, causing overestimated entropy rates and memory usage.
Study adds memory effect to Solow-Swan model for more accurate economic growth modeling.
problem Inaccuracies in classical Solow-Swan model in capturing long-term dynamics.
method Introduced fractional calculus with Caputo derivative into Solow-Swan framework.
result Fractional-order model shows significant impact on capital accumulation and stability.
We utilize long-term memory, fractal dimension and approximate entropy as input variables for the Efficiency Index [Kristoufek & Vosvrda (2013), Physica A 392]. This way, we are able to comment on stock market efficiency after controlling for different types of inefficiencies. Applying the methodology on 38 stock marke…
A new memory-based fusion layer improves multi-modal deep learning performance.
problem Improving performance of multi-modal deep learning by addressing long-term dependencies.
method Introducing a Memory based Attentive Fusion (MBAF) layer that incorporates both current and long-term dependencies.
result The MBAF layer enhances fusion and improves performance across different modalities and networks.
New CTRW model with memory explains long-term return autocorrelation.
problem Explaining long-term autocorrelation in financial returns.
method Proposed a Directed Continuous-Time Random Walk (CTRW) model with memory, considering only positive jumps and dependence on previous jumps.
result Bid-ask bounce explains only a small fraction of the long-term autocorrelation in financial returns.
State-regularized RNNs improve interpretability and performance on long-term memory tasks.
problem RNNs struggle with long-term memory and lack of interpretability.
method Introduce a stochastic state transition mechanism to limit state transitions to a finite set.
result State-regularized RNNs perform better on tasks requiring long-term memory.
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
New method reduces memory usage in deep HRNNs by replacing gradient backpropagation with local losses.
problem Memory constraints in training deep hierarchical RNNs.
method Replace gradient backpropagation with locally computable losses in deep HRNNs.
result Memory requirements reduced by a factor exponential in hierarchy depth.
ARISE models efficient markets without periodogram or Gaussianity assumptions.
problem Mimicking and learning long-term memory in efficient markets.
method ARISE process using aperiodic spectrum estimation and infinite-sum function of known processes.
result ARISE process has mean-square convergence, consistency, and asymptotic normality without periodogram and Gaussianity assumptions.
New framework captures long-term decision dependence in online learning.
problem Long-term dependence on past decisions in online learning.
method Introduces Online Convex Optimization with Unbounded Memory (OCO-UMB) and p p p -effective memory capacity. result Proves O ( H p T ) O(\sqrt{H_p T}) O ( H p T ) upper bound on policy regret and matching lower bound. One of the core tasks in multi-view learning is to capture relations among views. For sequential data, the relations not only span across views, but also extend throughout the view length to form long-term intra-view and inter-view interactions. In this paper, we present a new memory augmented neural network model that…
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
Proposes RMN for learning long-term dependencies in feed-forward networks.
problem Complicated training of deep RNN architectures.
method Residual Memory Neural Network (RMN) with residual and time-delayed connections.
result RMN and BRMN outperform LSTM and BLSTM networks in learning long-term and hierarchical information.
A Kalman Optimiser improves continual learning in deep neural networks.
problem Learning new tasks sequentially without forgetting previously learned knowledge.
method Divides the neural network into long-term and short-term memory units.
result The method enables continual learning and adaptation without forgetting.
Memory affects the convergence of stochastic optimization methods.
problem The impact of memory on the convergence of stochastic optimization methods.
method Using stochastic differential equations (SDEs) to study the role of memory in gradient-based algorithms.
result A flexible discrete-time algorithm with better stability properties than classical momentum.
New attack method improves neural network security.
problem Vulnerability of deep neural networks to adversarial attacks.
method Ensemble adversarial attack with long-term gradient memory.
result Enhanced robustness against adversarial attacks.
We analyze long-term memory properties of hourly prices of electricity in the Czech Republic between 2009 and 2012. As the dynamics of the electricity prices is dominated by cycles -- mainly intraday and daily -- we opt for the detrended fluctuation analysis, which is well suited for such specific series. We find that …
The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks. However, RNN still have a limited capacity to manipulate long-term memory. To bypass this weakness the most successful application…
Study examines cryptocurrency behavior during and after the pandemic.
problem Impact of the pandemic on cryptocurrency long-term memory and volatility.
method Used wavelet-based Hurst exponent analysis on eleven important coins.
result Long-term memory of returns mildly affected during pandemic, but volatility suffered temporary impact.
Enhanced LSTM model learns complex temporal dependencies.
problem Modeling long-term dependencies in sequential data.
method Gamma-LSTM with hierarchical memory units and gates.
result Gamma-LSTM outperforms regular and stacked LSTMs in sequence prediction.
ANPyC combats forgetting by pruning and consolidating neural parameters.
problem Catastrophic forgetting in neural networks, especially with long-term tasks.
method Adversarial Neural Pruning and Synaptic Consolidation (ANPyC) to balance task-relevant and irrelevant parameters.
result ANPyC prevents forgetting while enabling efficient learning of multiple tasks.
Researchers have used many different methods to detect the possibility of long-term dependence (long memory) in stock market returns, but evidence is in general mixed. In this paper, three different tests, (namely Rescaled Range (R/S), its modified form, and the semi-parametric method (GPH)), in addition to a new appro…
SMT improves robotic long-horizon tasks by embedding and utilizing past observations.
problem Long-horizon tasks in partially observable environments require effective long-term memory.
method Scene Memory Transformer (SMT) embeds and uses attention to exploit spatio-temporal dependencies.
result SMT outperforms existing policies in visual navigation tasks.
Algorithm learns linear systems from partial observations with near-optimal rate.
problem Identifying linear dynamical systems from partial observations, especially those with long-term memory.
method Multi-scale low-rank approximation using SVD on Hankel matrices of increasing sizes, combined with Fourier domain concentration bounds.
result Near-optimal rate of $\widetilde O\left(\sqrt\frac{d}{T}
ight)$ in H 2 \mathcal{H}_2 H 2 error, with logarithmic dependence on memory length. We investigate the temporal correlations and multifractal nature of trading volume of 22 liquid stocks traded on the Shenzhen Stock Exchange in 2003. We find that the trading volume exhibit size-dependent non-universal long memory and multifractal nature. No crossover in the power-law dependence of the detrended fluctu…
MCRM improves LSTM-GRU memory by compactly nesting them.
problem Improving recurrent neural network memory for temporal sequence tasks.
method Introducing MCRM, a nested LSTM-GRU architecture with compact memory.
result MCRMs outperform existing architectures on specific tasks.