This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay mem…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Replay Memory”
Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. …
Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consid…
Improves sparse reinforcement learning efficiency with OYMB.
problem Sparse rewards hinder reinforcement learning performance.
method Introduces OYMB, a sampler for HER to control minibatch makeup.
result HER combined with OYMB leads to faster real goal completion.
Online reinforcement learning agents are currently able to process an increasing amount of data by converting it into a higher order value functions. This expansion of the information collected from the environment increases the agent's state space enabling it to scale up to a more complex problems but also increases t…
In this paper, we propose a dual memory structure for reinforcement learning algorithms with replay memory. The dual memory consists of a main memory that stores various data and a cache memory that manages the data and trains the reinforcement learning agent efficiently. Experimental results show that the dual memory …
A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.
problem Stabilizing model-free off-policy deep reinforcement learning with soft divergence.
method Representing past experiences as a QGraph, selecting a subgraph with favorable structure, and using lower bounds for temporal difference learning.
result QG-DDPG method is less prone to soft divergence and more robust to hyperparameters.
Algorithm creates synthetic experiences to enhance Deep Reinforcement Learning.
problem Lack of synthetic experiences in classic Experience Replay.
method Bootstraps synthetic experiences to augment the replay memory.
result Synthetic experiences can improve learning speed and performance.
Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Me…
Paper proposes a new method to select memory data for online class-incremental learning.
problem Selecting which buffered images to replay for online class-incremental learning.
method Adversarial Shapley value scoring method to preserve latent decision boundaries.
result Proposed ASER method provides competitive or improved performance compared to state-of-the-art methods.
Prototype-based generative replay framework for online continual regression.
problem Addressing the challenge of non-stationary data streams in regression tasks.
method Adaptive output-space discretization model for prototype-based generative replay.
result Reduces forgetting and provides more stable performance.
Sequential learning of tasks using gradient descent leads to an unremitting decline in the accuracy of tasks for which training data is no longer available, termed catastrophic forgetting. Generative models have been explored as a means to approximate the distribution of old tasks and bypass storage of real data. Here …
New insights into experience replay in RL algorithms.
problem Understanding the impact of replay capacity and replay ratio in Q-learning.
method Systematic and extensive analysis of experience replay in Q-learning methods, focusing on replay capacity and replay ratio.
result Greater replay capacity significantly improves performance for certain algorithms, while other techniques offer limited benefit.
D-CBRS manages memory for continual learning by accounting for intra-class diversity.
problem Forgetting in continual learning, especially with class-imbalanced data.
method D-CBRS introduces a novel approach to store instances in memory, considering intra-class diversity.
result D-CBRS outperforms existing methods on data sets with intra-class diversity.
We introduce a lifelong language learning setup where a model needs to learn from a stream of text examples without any dataset identifier. We propose an episodic memory model that performs sparse experience replay and local adaptation to mitigate catastrophic forgetting in this setup. Experiments on text classificatio…
This paper tackles federated incremental learning with dynamic memory allocation for improved model performance in non-IID data.
problem Catastrophic forgetting in federated healthcare systems with non-IID data.
method Dynamic memory allocation strategy based on data replay mechanism.
result Significant performance improvements in medical image datasets compared to baseline models.
A new method prioritizes and recycles experiences for better reinforcement learning.
problem Improving reinforcement learning efficiency by prioritizing and recycling experiences.
method Double-prioritized state-recycled (DPSR) experience replay.
result DPSR achieved state-of-the-art results in Atari games, outperforming original and prioritized methods.
X-DER improves continual learning by revising replay memory and learning unseen classes.
problem Deep networks forget when learning new tasks sequentially.
method Combines rehearsal and Knowledge Distillation, revising replay memory and learning unseen classes.
result X-DER outperforms state-of-the-art on various benchmarks.
Optimal CL requires perfect memory and is NP-hard.
problem Designing CL algorithms that perform reliably and avoid catastrophic forgetting.
method Theoretical approach to derive computational properties of optimal CL algorithms.
result Optimal CL algorithms generally solve an NP-hard problem and require perfect memory.
New method improves continual learning by anchoring past knowledge.
problem Catastrophic forgetting in continual learning.
method Bilevel optimization to update current task knowledge while keeping past task predictions.
result Improves accuracy and forgetting metrics compared to experience replay.
GMED edits stored examples to improve continual learning.
problem Catastrophic forgetting in task-free continual learning.
method Gradient-based memory editing of stored examples.
result GMED-edited examples help prevent forgetting.
Unified study of stateful replay for streaming learning, reducing forgetting by 2-3x.
problem Catastrophic forgetting in streaming generative and predictive learning.
method Unified analysis of stateful replay for autoencoding, forecasting, and classification tasks.
result Stateful replay reduces average forgetting by a factor of 2-3 on heterogeneous multi-task streams.
A new method for generating replay samples on the fly, optimizing for not forgetting.
problem Addressing the issue of forgetting in neural networks.
method Generates auxiliary samples on the fly using the model's implicit memory, specialized to each real training batch.
result Optimizing for not forgetting leads to more efficient and scalable generation of specialized samples.
Graph-based rehearsal improves continual learning performance.
problem Catastrophic forgetting in continual learning models.
method Augmenting a memory array with a learnable random graph to capture pairwise similarities.
result Our model consistently outperforms baselines on task-free continual learning benchmarks.
Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behavio…
Our research is focused on understanding and applying biological memory transfers to new AI systems that can fundamentally improve their performance, throughout their fielded lifetime experience. We leverage current understanding of biological memory transfer to arrive at AI algorithms for memory consolidation and repl…
Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the ba…
Improved Politex algorithm reduces regret bound to O(√T) with experience replay.
problem Learning in infinite-horizon MDPs with function approximation.
method Sharpened regret analysis of Politex algorithm, experience replay implementation.
result First high-probability O(√T) regret bound for computationally efficient algorithm.
Modern Hopfield networks help prevent forgetting in generative models after task changes.
problem How to prevent forgetting in generative models after task changes.
method Introduce intrinsic forgetting as an increase in Hopfield energy after task change, analyze memory replay effectiveness, and validate predictions in experiments.
result High-energy, outlier-like samples are more forgettable than cluster-like samples, and energy-based selection of replay samples mitigates forgetting.
The brain optimizes memory by forgetting what's predictable, improving generalization.
problem Memory consolidation struggles with representational drift, semanticisation, and offline replay.
method Proposes predictive forgetting as a mechanism to optimize generalization by reducing complexity.
result Predictive forgetting improves information-theoretic generalization bounds on stored representations.
LiDER refreshes past experiences in RL by dreaming about them.
problem Improving data efficiency in off-policy RL algorithms.
method Refreshing past experiences in a replay buffer using the current policy.
result LiDER consistently improves performance in Atari games.
Study improves neural network performance in sequential learning for image classification.
problem Improving neural network performance in sequential learning for image classification.
method Evaluation of approaches for computing prequential description lengths, proposing forward-calibration and replay-streams.
result Improved description lengths for image classification datasets, outperforming previous results.
TOMA generates abstract graphs for RL, reducing memory and computation costs.
problem High memory and computation costs in graph generation for RL.
method Topological Map Abstraction (TOMA) for generating abstract graphs.
result TOMA reduces memory and computation costs compared to existing methods.
New analysis shows FM learns underlying dynamical structure, not just trajectory replay.
problem Understanding whether flow matching models learn transferable dynamical structure or merely replay trajectories.
method Derived velocity field implied by FM objective, characterized as a continuous-time dynamical system.
result FM models can be seen as parametric surrogates of nonparametric solutions, providing strong probabilistic forecasts.
MER algorithm speeds up VI solving with Markovian data.
problem Solving stochastic variational inequalities with Markovian data.
method MER algorithm using multi-scale sampling from a Markovian buffer.
result Achieves faster convergence without knowing Markov chain mixing time.
Modern deep reinforcement learning methods have departed from the incremental learning required for eligibility traces, rendering the implementation of the -return difficult in this context. In particular, off-policy methods that utilize experience replay remain problematic because their random sampling of minibatch…
Neural network tackles continual learning with neuromodulation and local error signals.
problem Catastrophic forgetting in continuous learning.
method Biologically-inspired neural architecture with local learning and neuromodulation, combined with transfer metalearning.
result Superior performance in continual learning tasks compared to other approaches.
A simple technique improves continual learning by 50% on image datasets.
problem Challenges in training neural networks on a stream of shifting data.
method Experience Replay (ER) with five tricks to mitigate its shortcomings.
result ER, enhanced with tricks, achieves significant accuracy gains.
This work tackles online memory selection in continual learning using information theory.
problem Online selection of a representative replay memory from data streams.
method Information-theoretic criteria (surprise, learnability) and Bayesian model for efficient computation.
result InfoRS improves robustness against data imbalance compared to reservoir sampling.
A new method for continual learning in GANs learns new modes with limited data.
problem Learning new target modes with limited samples while preserving previously learned ones.
method Mode-affinity score for generative modeling, generator replay, and weighted label generation.
result Gains over state-of-the-art methods, even with fewer training samples.
We first pose the Unsupervised Progressive Learning (UPL) problem: an online representation learning problem in which the learner observes a non-stationary and unlabeled data stream, learning a growing number of features that persist over time even though the data is not stored or replayed. To solve the UPL problem we …
A novel approach stores encoded images as centroids and covariance matrices to improve classification accuracy with less memory.
problem Catastrophic forgetting and memory limitations in continual learning.
method Trains autoencoders with Neural Style Transfer to encode images, replay encoded episodes to avoid forgetting, and use centroids and covariance matrices for pseudo-images when memory is full.
result Increases classification accuracy by 13-17% over state-of-the-art methods on benchmark datasets, while requiring 78% less storage space.
In continual learning (CL), an agent learns from a stream of tasks leveraging prior experience to transfer knowledge to future tasks. It is an ideal framework to decrease the amount of supervision in the existing learning algorithms. But for a successful knowledge transfer, the learner needs to remember how to perform …
Optimizer memory affects learning rate sensitivity in shuffle order, impacting fine-tuning noise.
problem Optimizer memory affects the learning rate sensitivity in shuffle order, leading to fine-tuning noise.
method Isolated the mechanism of fixed-clock optimizer memory affecting the learning rate sensitivity in shuffle order, deriving a fit-free way to size the noise.
result Fixed-clock optimizers like AdamW produce a larger first-order noise channel compared to memoryless optimizers, affecting fine-tuning comparisons.
New modifiers improve noisy RNN replay in hippocampal networks.
problem Improving noisy RNN replay in hippocampal networks.
method Three approaches: hidden state leakage, adaptation, and momentum.
result Hidden state leakage, adaptation, and momentum improve noisy RNN replay.
This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to th…
We present a new replay-based method of continual classification learning that we term "conditional replay" which generates samples and labels together by sampling from a distribution conditioned on the class. We compare conditional replay to another replay-based continual learning paradigm (which we term "marginal rep…
Paper improves online time series forecasting by combining natural gradient and robust t-distribution.
problem Online time series forecasting challenges in rapidly adapting to evolving data.
method Reframed neural network optimization as a parameter filtering problem, using natural gradient and Student's t likelihood.
result Natural Score-driven Replay (NatSR) achieves stronger forecasting performance than state-of-the-art methods.