Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consid…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Experience Replay”
This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to th…
This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay mem…
New insights into experience replay in RL algorithms.
problem Understanding the impact of replay capacity and replay ratio in Q-learning.
method Systematic and extensive analysis of experience replay in Q-learning methods, focusing on replay capacity and replay ratio.
result Greater replay capacity significantly improves performance for certain algorithms, while other techniques offer limited benefit.
A new method prioritizes and recycles experiences for better reinforcement learning.
problem Improving reinforcement learning efficiency by prioritizing and recycling experiences.
method Double-prioritized state-recycled (DPSR) experience replay.
result DPSR achieved state-of-the-art results in Atari games, outperforming original and prioritized methods.
ReaPER improves learning efficiency by prioritizing reliable experiences.
problem Inefficient sampling of past experiences in reinforcement learning.
method Introducing a novel measure of reliability to prioritize experiences in PER.
result ReaPER outperforms PER in various environments, including Atari-10.
The paper improves reinforcement learning stability and efficiency with a new theoretical framework.
problem Stability and efficiency in reinforcement learning, especially in data-scarce scenarios.
method Theoretical framework using resampled - and -statistics to model experience replay, applied to policy evaluation and kernel ridge regression.
result Significant improvements in stability and efficiency, particularly in data-scarce scenarios.
Curious Replay improves model-based reinforcement learning agents' adaptability.
problem Existing model-based reinforcement learning agents struggle to adapt quickly to changing environments.
method Curious Replay uses a curiosity-based priority signal for prioritized experience replay tailored to model-based agents.
result Agents using Curious Replay achieve improved performance in exploration and on benchmarks.
Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized experience replay (PER) which samples important transitions more frequently. In this paper, we propo…
An important component of many Deep Reinforcement Learning algorithms is the Experience Replay which serves as a storage mechanism or memory of made experiences. These experiences are used for training and help the agent to stably find the perfect trajectory through the problem space. The classic Experience Replay howe…
We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ …
Our work is a simple extension of the paper "Exploration by Random Network Distillation". More in detail, we show how to efficiently combine Intrinsic Rewards with Experience Replay in order to achieve more efficient and robust exploration (with respect to PPO/RND) and consequently better results in terms of agent perf…
LiDER refreshes past experiences in RL by dreaming about them.
problem Improving data efficiency in off-policy RL algorithms.
method Refreshing past experiences in a replay buffer using the current policy.
result LiDER consistently improves performance in Atari games.
Generative replay improves continual learning by using generated data as negative examples.
problem Catastrophic forgetting in continual learning.
method Using generative models to provide negative examples for new classes.
result Generative replay can improve learning new classes even when existing approaches fail.
Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behavio…
Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. …
ER-GNN uses experience replay to prevent GNNs from forgetting previous tasks.
problem Catastrophic forgetting in GNNs when learning multiple tasks sequentially.
method Experience Replay framework to store and replay knowledge from previous tasks.
result ER-GNN effectively mitigates catastrophic forgetting in GNNs.
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
RER improves sample complexity by updating in reverse order.
problem Theoretical analysis limits RER's convergence rate.
method Tighter analysis for larger learning rates and longer sequences.
result RER converges faster with larger learning rates and longer sequences.
Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle…
In this paper, we propose a replay attack spoofing detection system for automatic speaker verification using multitask learning of noise classes. We define the noise that is caused by the replay attack as replay noise. We explore the effectiveness of training a deep neural network simultaneously for replay attack spoof…
NERS improves RL by sampling diverse transitions considering local and global contexts.
problem Sampling biases in experience replay lead to redundant transitions.
method Neural Experience Replay Sampler (NERS) that considers both local and global contexts.
result NERS significantly improves RL performance by sampling diverse and meaningful transitions.
Improved Politex algorithm reduces regret bound to O(√T) with experience replay.
problem Learning in infinite-horizon MDPs with function approximation.
method Sharpened regret analysis of Politex algorithm, experience replay implementation.
result First high-probability O(√T) regret bound for computationally efficient algorithm.
VRER selectively reuses samples to improve policy optimization in complex systems.
problem Lack of effective reuse of historical samples in reinforcement learning.
method Variance reduction based experience replay (VRER) framework.
result VRER accelerates policy optimization and enhances performance.
SynthER uses generative models to augment limited RL experience.
problem Limited data for reinforcement learning agents.
method SynthER leverages diffusion models to generate synthetic experience data.
result SynthER significantly improves sample efficiency and training of RL agents.
A distributed system identification method for LTI systems using reverse experience replay.
problem Online system identification of LTI systems over multi-agent networks.
method DSGD-RER, a distributed variant of SGD-RER with backward updates.
result The estimation error decreases as the network size grows.
This paper presents a method to improve continual learning stability and plasticity.
problem Balancing learning stability and plasticity in deep learning.
method Batch-level Experience Replay with Review approach.
result Achieved 1st place in all three scenarios of the CVPR 2020 CLVision challenge.
Modern Hopfield networks help prevent forgetting in generative models after task changes.
problem How to prevent forgetting in generative models after task changes.
method Introduce intrinsic forgetting as an increase in Hopfield energy after task change, analyze memory replay effectiveness, and validate predictions in experiments.
result High-energy, outlier-like samples are more forgettable than cluster-like samples, and energy-based selection of replay samples mitigates forgetting.
Improves sparse reinforcement learning efficiency with OYMB.
problem Sparse rewards hinder reinforcement learning performance.
method Introduces OYMB, a sampler for HER to control minibatch makeup.
result HER combined with OYMB leads to faster real goal completion.
New method improves deep RL by combining emphatic weightings with replay data.
problem Improving sample efficiency and scaling model-free RL methods.
method Developed a multi-step emphatic weighting and time-reversed -step TD learning algorithm.
result The new approach reduces variance and provides convergence guarantees.
Online reinforcement learning agents are currently able to process an increasing amount of data by converting it into a higher order value functions. This expansion of the information collected from the environment increases the agent's state space enabling it to scale up to a more complex problems but also increases t…
New loss function equivalence reveals PER's uniform sampling can be improved.
problem Improving Prioritized Experience Replay (PER) for better learning efficiency.
method Transforming non-uniformly sampled data loss functions into uniformly sampled ones.
result Some environments can replace PER with a new loss function without performance loss.
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of exper…
Modern deep reinforcement learning methods have departed from the incremental learning required for eligibility traces, rendering the implementation of the -return difficult in this context. In particular, off-policy methods that utilize experience replay remain problematic because their random sampling of minibatch…
Policy gradient reinforcement learning (RL) algorithms have achieved impressive performance in challenging learning tasks such as continuous control, but suffer from high sample complexity. Experience replay is a commonly used approach to improve sample efficiency, but gradient estimators using past trajectories typica…
A new method improves continual learning by replaying pseudo data and using orthogonal weight modification.
problem Catastrophic forgetting in class incremental learning.
method Generative replay with orthogonal weight modification.
result Our method achieves substantial improvement over conventional generative replay and OWM.
Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising way to train an agent to perform goal-directed tasks is to use Hindsight Learning approaches. In thes…
A simple technique improves continual learning by 50% on image datasets.
problem Challenges in training neural networks on a stream of shifting data.
method Experience Replay (ER) with five tricks to mitigate its shortcomings.
result ER, enhanced with tricks, achieves significant accuracy gains.
We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan…
D2D-SPL uses discrete states and a classifier to train RL faster.
problem Training neural networks in RL due to correlated samples.
method Discretizes state space, uses actor-critic, selects input/target pairs, trains classifier.
result Trains faster than state-of-the-art methods.
In Hindsight Experience Replay (HER), a reinforcement learning agent is trained by treating whatever it has achieved as virtual goals. However, in previous work, the experience was replayed at random, without considering which episode might be the most valuable for learning. In this paper, we develop an energy-based fr…
Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Me…
New algorithm optimizes linear system estimation from single trajectory.
problem Estimating LTI systems from a single trajectory.
method SGD with Reverse Experience Replay ()
result Optimal guarantees for parameter and prediction errors.
New algorithm combines curriculum learning with HER for complex object manipulation tasks.
problem Learning complex sequential object manipulation tasks from scratch is challenging.
method Curriculum learning with Hindsight Experience Replay (HER) for recurrent object manipulation tasks.
result Significant improvement in learning sequential object manipulation tasks compared to vanilla-HER.
ReF-ER algorithm improved performance in multi-agent reinforcement learning.
problem Improving performance in multi-agent reinforcement learning environments.
method Extended ReF-ER algorithm to include dependencies between agents and modeled environment dynamics.
result ReF-ER MARL outperforms state-of-the-art algorithms in collaborative environments.
A new buffer system improves continual learning in RL agents by adapting to changing environments.
problem Improving RL agents' ability to learn from changing environments over time.
method Multi-timescale replay buffer combined with invariant risk minimization.
result The method shows improvement over baselines in continual learning settings.
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language underst…