Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consid…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “replay”
New modifiers improve noisy RNN replay in hippocampal networks.
problem Improving noisy RNN replay in hippocampal networks.
method Three approaches: hidden state leakage, adaptation, and momentum.
result Hidden state leakage, adaptation, and momentum improve noisy RNN replay.
New insights into experience replay in RL algorithms.
problem Understanding the impact of replay capacity and replay ratio in Q-learning.
method Systematic and extensive analysis of experience replay in Q-learning methods, focusing on replay capacity and replay ratio.
result Greater replay capacity significantly improves performance for certain algorithms, while other techniques offer limited benefit.
This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to th…
We present a new replay-based method of continual classification learning that we term "conditional replay" which generates samples and labels together by sampling from a distribution conditioned on the class. We compare conditional replay to another replay-based continual learning paradigm (which we term "marginal rep…
This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay mem…
Curious Replay improves model-based reinforcement learning agents' adaptability.
problem Existing model-based reinforcement learning agents struggle to adapt quickly to changing environments.
method Curious Replay uses a curiosity-based priority signal for prioritized experience replay tailored to model-based agents.
result Agents using Curious Replay achieve improved performance in exploration and on benchmarks.
A new method prioritizes and recycles experiences for better reinforcement learning.
problem Improving reinforcement learning efficiency by prioritizing and recycling experiences.
method Double-prioritized state-recycled (DPSR) experience replay.
result DPSR achieved state-of-the-art results in Atari games, outperforming original and prioritized methods.
ReaPER improves learning efficiency by prioritizing reliable experiences.
problem Inefficient sampling of past experiences in reinforcement learning.
method Introducing a novel measure of reliability to prioritize experiences in PER.
result ReaPER outperforms PER in various environments, including Atari-10.
Unified study of stateful replay for streaming learning, reducing forgetting by 2-3x.
problem Catastrophic forgetting in streaming generative and predictive learning.
method Unified analysis of stateful replay for autoencoding, forecasting, and classification tasks.
result Stateful replay reduces average forgetting by a factor of 2-3 on heterogeneous multi-task streams.
In this paper, we propose a replay attack spoofing detection system for automatic speaker verification using multitask learning of noise classes. We define the noise that is caused by the replay attack as replay noise. We explore the effectiveness of training a deep neural network simultaneously for replay attack spoof…
Generative replay improves continual learning by using generated data as negative examples.
problem Catastrophic forgetting in continual learning.
method Using generative models to provide negative examples for new classes.
result Generative replay can improve learning new classes even when existing approaches fail.
The paper improves reinforcement learning stability and efficiency with a new theoretical framework.
problem Stability and efficiency in reinforcement learning, especially in data-scarce scenarios.
method Theoretical framework using resampled - and -statistics to model experience replay, applied to policy evaluation and kernel ridge regression.
result Significant improvements in stability and efficiency, particularly in data-scarce scenarios.
A new method improves continual learning by replaying pseudo data and using orthogonal weight modification.
problem Catastrophic forgetting in class incremental learning.
method Generative replay with orthogonal weight modification.
result Our method achieves substantial improvement over conventional generative replay and OWM.
Our work is a simple extension of the paper "Exploration by Random Network Distillation". More in detail, we show how to efficiently combine Intrinsic Rewards with Experience Replay in order to achieve more efficient and robust exploration (with respect to PPO/RND) and consequently better results in terms of agent perf…
Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized experience replay (PER) which samples important transitions more frequently. In this paper, we propo…
Prototype-based generative replay framework for online continual regression.
problem Addressing the challenge of non-stationary data streams in regression tasks.
method Adaptive output-space discretization model for prototype-based generative replay.
result Reduces forgetting and provides more stable performance.
New framework models echo chamber learning, proving tight bounds on algorithm performance.
problem Echo chambers in machine learning where systems learn from self-annotated data.
method Online Learning in the Replay Setting, Extended Threshold dimension, closure-based learner.
result Proves tight bounds on algorithm performance against replay adversaries.
We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ …
Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. …
An important component of many Deep Reinforcement Learning algorithms is the Experience Replay which serves as a storage mechanism or memory of made experiences. These experiences are used for training and help the agent to stably find the perfect trajectory through the problem space. The classic Experience Replay howe…
Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behavio…
LiDER refreshes past experiences in RL by dreaming about them.
problem Improving data efficiency in off-policy RL algorithms.
method Refreshing past experiences in a replay buffer using the current policy.
result LiDER consistently improves performance in Atari games.
RER improves sample complexity by updating in reverse order.
problem Theoretical analysis limits RER's convergence rate.
method Tighter analysis for larger learning rates and longer sequences.
result RER converges faster with larger learning rates and longer sequences.
Modern Hopfield networks help prevent forgetting in generative models after task changes.
problem How to prevent forgetting in generative models after task changes.
method Introduce intrinsic forgetting as an increase in Hopfield energy after task change, analyze memory replay effectiveness, and validate predictions in experiments.
result High-energy, outlier-like samples are more forgettable than cluster-like samples, and energy-based selection of replay samples mitigates forgetting.
Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Me…
Improves sparse reinforcement learning efficiency with OYMB.
problem Sparse rewards hinder reinforcement learning performance.
method Introduces OYMB, a sampler for HER to control minibatch makeup.
result HER combined with OYMB leads to faster real goal completion.
ER-GNN uses experience replay to prevent GNNs from forgetting previous tasks.
problem Catastrophic forgetting in GNNs when learning multiple tasks sequentially.
method Experience Replay framework to store and replay knowledge from previous tasks.
result ER-GNN effectively mitigates catastrophic forgetting in GNNs.
We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan…
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
Improved Politex algorithm reduces regret bound to O(√T) with experience replay.
problem Learning in infinite-horizon MDPs with function approximation.
method Sharpened regret analysis of Politex algorithm, experience replay implementation.
result First high-probability O(√T) regret bound for computationally efficient algorithm.
New method improves deep RL by combining emphatic weightings with replay data.
problem Improving sample efficiency and scaling model-free RL methods.
method Developed a multi-step emphatic weighting and time-reversed -step TD learning algorithm.
result The new approach reduces variance and provides convergence guarantees.
Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle…
Online reinforcement learning agents are currently able to process an increasing amount of data by converting it into a higher order value functions. This expansion of the information collected from the environment increases the agent's state space enabling it to scale up to a more complex problems but also increases t…
A new buffer system improves continual learning in RL agents by adapting to changing environments.
problem Improving RL agents' ability to learn from changing environments over time.
method Multi-timescale replay buffer combined with invariant risk minimization.
result The method shows improvement over baselines in continual learning settings.
An attacker may use a variety of techniques to fool an automatic speaker verification system into accepting them as a genuine user. Anti-spoofing methods meanwhile aim to make the system robust against such attacks. The ASVspoof 2017 Challenge focused specifically on replay attacks, with the intention of measuring the …
A major obstacle to developing artificial intelligence applications capable of true lifelong learning is that artificial neural networks quickly or catastrophically forget previously learned tasks when trained on a new one. Numerous methods for alleviating catastrophic forgetting are currently being proposed, but diffe…
A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.
problem Stabilizing model-free off-policy deep reinforcement learning with soft divergence.
method Representing past experiences as a QGraph, selecting a subgraph with favorable structure, and using lower bounds for temporal difference learning.
result QG-DDPG method is less prone to soft divergence and more robust to hyperparameters.
VRER selectively reuses samples to improve policy optimization in complex systems.
problem Lack of effective reuse of historical samples in reinforcement learning.
method Variance reduction based experience replay (VRER) framework.
result VRER accelerates policy optimization and enhances performance.
Automatic speaker verification systems are vulnerable to audio replay attacks which bypass security by replaying recordings of authorized speakers. Replay attack detection (RA) detection systems built upon Residual Neural Networks (ResNet)s have yielded astonishing results on the public benchmark ASVspoof 2019 Physical…
In Multi-Goal Reinforcement Learning, an agent learns to achieve multiple goals with a goal-conditioned policy. During learning, the agent first collects the trajectories into a replay buffer, and later these trajectories are selected randomly for replay. However, the achieved goals in the replay buffer are often biase…
Develops a support-aware framework for reserve-policy selection in advertising markets.
problem Log-based reserve-price evaluation risks weak support and subgroup harm.
method Support-aware offline decision framework converting logged evidence into certified policies.
result Preserves the best gate-passing policy while eliminating only policies with certified regret.
Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-p…
NERS improves RL by sampling diverse transitions considering local and global contexts.
problem Sampling biases in experience replay lead to redundant transitions.
method Neural Experience Replay Sampler (NERS) that considers both local and global contexts.
result NERS significantly improves RL performance by sampling diverse and meaningful transitions.
New approach prevents model collapse in language generation.
problem Model collapse risk in large language models.
method Introduces a replay adversary to study language generation.
result Replay limits generation in certain ways but not all.
Sparse reward problems are one of the biggest challenges in Reinforcement Learning. Goal-directed tasks are one such sparse reward problems where a reward signal is received only when the goal is reached. One promising way to train an agent to perform goal-directed tasks is to use Hindsight Learning approaches. In thes…
This paper presents a method to improve continual learning stability and plasticity.
problem Balancing learning stability and plasticity in deep learning.
method Batch-level Experience Replay with Review approach.
result Achieved 1st place in all three scenarios of the CVPR 2020 CLVision challenge.
Modern deep reinforcement learning methods have departed from the incremental learning required for eligibility traces, rendering the implementation of the -return difficult in this context. In particular, off-policy methods that utilize experience replay remain problematic because their random sampling of minibatch…