P3O merges on-policy and off-policy updates without extra hyper-parameters.
problem Combining on-policy and off-policy RL algorithms to reduce sample complexity.
method Interleaves off-policy updates with on-policy updates using effective sample size.
result P3O reduces sample complexity of state-of-the-art algorithms.
MPPO improves exploration efficiency for on-policy methods.
problem Inefficient exploration in on-policy methods.
method MPPO uses a population of diverse policies to enable better exploration.
result MPPO significantly outperforms state-of-the-art exploration methods.
A new approach to meta-RL reduces sample inefficiency.
problem Meta-RL requires impractical amounts of on-policy experience.
method Federated learning approach where off-policy learners solve individual tasks and consolidate solutions into a meta-learner.
result Significant improvements in meta-RL sample efficiency.
Study investigates key design choices in on-policy RL algorithms.
problem Lack of transparency in RL algorithm implementations.
method Implemented >50 design choices in a unified RL framework.
result Insights and recommendations for on-policy RL training.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
Improves on-policy RL by reusing data from multiple policies.
problem Lack of reuse of data from previous policies in on-policy RL.
method Adapts replay buffer concept to combine on- and off-policy learning.
result Method outperforms state-of-the-art on-policy RL algorithms.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.
problem Combining robust on-policy methods with efficient offline data for hybrid RL.
method Integrates off-policy training on offline data into on-policy NPG framework.
result Achieves state-of-the-art theoretical guarantees and maintains on-policy NPG guarantees.
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
CODA resolves coordination issues in offline multi-agent reinforcement learning.
problem Coordination failure in offline multi-agent reinforcement learning.
method Diffusion-based multi-agent trajectory generator for data augmentation.
result CODA resolves coordination pathologies in continuous polynomial games and complex benchmarks.
This work analyzes the gap between off-policy and on-policy policy gradient methods and provides conditions to reduce this gap.
problem The gap between off-policy and on-policy policy gradient methods and conditions to reduce it.
method Theoretical analysis and empirical evidence of conditions to reduce the on-off gap.
result Conditions to reduce the on-off gap between off-policy and on-policy policy gradient methods.
This thesis tackles policy learning in large action spaces using on-policy and off-policy methods.
problem Challenges in learning with large action spaces, including inefficient exploration, sparse data, high-variance importance weights, and difficult optimization.
method Developed structured Bayesian methods for on-policy learning and structured direct methods for off-policy learning.
result Introduced meTS and dTS for on-policy learning, and sDM for off-policy learning, with regret guarantees and optimization strategies.
Reinforcement learning agents need exploratory behaviors to escape from local optima. These behaviors may include both immediate dithering perturbation and temporally consistent exploration. To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for t…
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
problem The cost of fully online on-policy distillation for multi-turn interactions.
method ReOPD, an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes, addressing the prefix trap and distribution shift.
result ReOPD preserves or improves OPD-level accuracy, uses zero tool calls, and is at least 4imes faster per training step. RPO uses past and future state-action info for better policy optimization.
problem Sample inefficiency in on-policy reinforcement learning methods.
method Reflective Policy Optimization (RPO) integrates past and future state-action info for policy improvement.
result RPO improves policy performance and contracts the solution space, leading to faster convergence.
New reinforcement learning algorithms improve policy optimization with entropy regularization.
problem Improving policy optimization in reinforcement learning.
method Soft policy gradient theorem (SPGT) and new policy optimization algorithms.
result New algorithms outperform prior works on various benchmark tasks.
We study reinforcement learning of chatbots with recurrent neural network architectures when the rewards are noisy and expensive to obtain. For instance, a chatbot used in automated customer service support can be scored by quality assurance agents, but this process can be expensive, time consuming and noisy. Previous …
Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a partic…
New RL algorithms improve average-reward performance.
problem Improving RL algorithms for average-reward criteria.
method Developed novel algorithms addressing average-reward criterion directly.
result ATRPO significantly outperforms TRPO in challenging MuJuCo environments.
Efficient actor-critic learning with shared experience replay improves data efficiency.
problem Challenges in actor-critic reinforcement learning with experience replay and off-policy learning stability.
method Combining actor-critic algorithms with shared experience replay, analyzing V-trace, proposing a trust region scheme.
result State-of-the-art data efficiency on Atari achieved with 200M environment frames.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
Proposes a Riemannian optimization for policy improvement in MDPs.
problem Optimizing policy functions in Markov decision processes (MDPs).
method Riemannian proximal optimization algorithm with Gaussian mixture model (GMM).
result Guaranteed convergence and efficacy demonstrated in preliminary experiments.
Study evaluates reinforcement learning for trading diverse stocks, finds Q-learning outperforms.
problem Evaluating reinforcement learning for trading diverse stocks.
method Implemented Value Iteration (VI), State-action-reward-state-action (SARSA), and Q-Learning on a diverse stock portfolio dataset.
result Q-learning performs better than VI and SARSA during testing, but performance varies based on market conditions.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this p…
We consider the generic approach of using an experience memory to help exploration by adapting a restart distribution. That is, given the capacity to reset the state with those corresponding to the agent's past observations, we help exploration by promoting faster state-space coverage via restarting the agent from a mo…
The paper introduces a novel method for stable off-policy learning using value function chaining.
problem Stability issues in off-policy reinforcement learning.
method The approach involves learning on-policy first, then chaining off-policy value estimates.
result The method guarantees convergence and can approximate off-policy TD solutions.
This paper merges deterministic policy gradient estimations to improve deep reinforcement learning performance.
problem The bias-variance tradeoff in estimating and using policy gradients for deep reinforcement learning.
method Introduces elite policy gradients and a two-step merging method to balance bias-variance tradeoffs.
result Two-step merging outperforms interpolation merging and state-of-the-art algorithms on benchmark control tasks.
New estimator reduces bias-variance tradeoff in Markovian interference experiments.
problem Estimating impact of interventions in systems with limited resources.
method Differences-In-Q (DQ) estimator for on-policy policy evaluation.
result DQ estimator has exponentially smaller variance than off-policy methods.
We generalise the problem of inverse reinforcement learning to multiple tasks, from multiple demonstrations. Each one may represent one expert trying to solve a different task, or as different experts trying to solve the same task. Our main contribution is to formalise the problem as statistical preference elicitation,…
Algorithm learns new tasks efficiently from past experience.
problem Lack of robustness to distributional shift in meta-reinforcement learning.
method Model Identification and Experience Relabeling (MIER) using dynamics models.
result Efficient extrapolation to out-of-distribution tasks.
Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization…
Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evalua…
Meta-RL algorithm improves sample efficiency and adaptability.
problem Challenges in meta-reinforcement learning, especially on-policy experience and task uncertainty.
method Develops an off-policy meta-RL algorithm that probabilistically infers task variables.
result Significantly outperforms prior algorithms in sample efficiency and asymptotic performance.
MO2 learns useful behaviours from past experience for new tasks.
problem Discovering useful behaviours from past experience and transferring them to new tasks.
method Model-Based Offline Options (MO2) framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces.
result MO2 outperforms recent option learning methods on complex long-horizon continuous control tasks.
Unified framework for reliable uncertainty quantification in RL.
problem Uncertainty quantification in high-stakes reinforcement learning.
method Unified conformal prediction framework integrating distributional RL and conformal calibration.
result Significantly improved coverage and reliability over standard methods.
Swift-Sarsa combines TD learning with Sarsa to control tasks robustly.
problem Learning effective control policies from noisy signals.
method Combines True Online Sarsa(λ) with step-size optimization and decay. result Swift-Sarsa learns relevant signals without prior knowledge.
Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints on the long-term cost. A canonical approach for solving CMDPs is the primal-dual method which updat…
Extends RL to random stopping times, improving optimization.
problem Real-world applications with random stopping times.
method Extended RL framework to random stopping times, derived new formulas.
result Improves optimization convergence with new formulas.
DiCE uses diverse agents to explore and learn, avoiding local minima.
problem Local minima in RL due to limited exploration and correlated behavior.
method DiCE employs a group of heterogeneous agents to explore simultaneously and share experiences, with a diversity regularization mechanism.
result DiCE achieves substantial improvement over baselines in MuJoCo locomotion tasks.
Improves reinforcement learning stability and efficiency.
problem Combining stability and efficiency in reinforcement learning.
method Combines on-policy stability with off-policy sample reuse.
result Demonstrates improved performance in both theory and practice.
Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, an…
In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theore…
Policy gradient aims to maximize expected return using gradient ascent.
problem Finding a policy that maximizes expected return in a given class of policies.
method Gradient ascent applied to a differentiable model of the policy, estimating the gradient of expected return.
result Policy gradient methods require on-policy data for gradient estimation, limiting sample efficiency.
A new machine learning approach for generating high-quality chordal extensions.
problem Defining the definitive relation between chordal extension and optimization algorithm performance.
method On-policy imitation learning scheme mimicking the minimum degree rule to generate high-quality chordal extensions.
result On-policy imitation learning approach effectively learns the minimum degree policy and produces graphs with desirable fill-in characteristics.
Proposes log density gradient to improve reinforcement learning sample complexity.
problem Residual error in gradient estimation in policy gradient methods.
method Log density gradient method to correct residual error, using state-action discounted distributional formulation.
result Min-max optimization method to approximate log density gradient with on-policy samples, achieving sample complexity of m−1/2. Develops RL algorithm for lifelong non-stationary environments.
problem Challenges of reinforcement learning in environments with persistent change.
method Formalizes lifelong non-stationarity, uses latent variable models, and leverages online learning and probabilistic inference.
result Substantial improvement in performance over non-reasoning approaches in lifelong non-stationary environments.
PCHID improves sample efficiency in reinforcement learning tasks.
problem Sparse rewards make learning policies difficult in reinforcement learning.
method Hindsight Inverse Dynamics with Hindsight Experience Replay and Policy Continuation.
result PCHID significantly improves sample efficiency and final performance on multi-goal tasks.