arXiv research

STORM-PG uses momentum for faster policy gradient updates.

The paper interprets policy-gradient algorithms using continuation theory.

The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The …

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our…

Optimizes Thompson sampling policies using policy gradient methods.

Paper tackles efficient policy gradient estimation from off-policy data.

Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…

Paper proposes an $ε$-policy gradient for online pricing, reducing regret to $\mathcal{O}(\sqrt{T})$.

Memory-efficient algorithm reduces variance in off-policy RL.

Policy gradient algorithm with variable learning rates achieves near-optimal performance in multi-arm bandit problems.

Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradien…

MAGE optimizes policies using action gradients from model-based learning.

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…

Accelerates policy optimization in RL with optimistic and adaptive updates.

This paper studies GAIL's global convergence for general MDP and nonlinear rewards.

Short note on soft-max and policy gradients in bandit problems using Lyapunov functions.

Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …

Paper proposes a policy gradient method for confounded POMDPs.

Reinforcement learning algorithms such as the deep deterministic policy gradient algorithm (DDPG) has been widely used in continuous control tasks. However, the model-free DDPG algorithm suffers from high sample complexity. In this paper we consider the deterministic value gradients to improve the sample efficiency of …

Study policy gradient and actor-critic methods for continuous-time reinforcement learning.

Improves RL efficiency with nonparametric methods.

Large deviations theory applied to policy gradient methods.

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

Study shows code-level optimizations significantly impact deep RL algorithms.

Entropy regularization improves policy optimization in reinforcement learning.

Reinforcement learning is a promising approach to learning robotics controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomen…

A quantum reinforcement learning algorithm reduces sample complexity.

The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.

Developed policy gradient methods for stochastic control with exit time, outperforming traditional techniques in share repurchase pricing.

Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies

DBQPG improves policy gradient estimation with fewer samples.

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

Policy optimization is a core component of reinforcement learning (RL), and most existing RL methods directly optimize parameters of a policy based on maximizing the expected total reward, or its surrogate. Though often achieving encouraging empirical success, its underlying mathematical principle on {\em policy-distri…

Paper proposes MCTSPO for better reinforcement learning policy optimization.

We study a reinforcement learning setting, where the state transition function is a convex combination of a stochastic continuous function and a deterministic function. Such a setting generalizes the widely-studied stochastic state transition setting, namely the setting of deterministic policy gradient (DPG). We firstl…

Protects proprietary policies from imitation learning by training adversarial policy ensembles.

Paper proposes FPG algorithm for unbiased off-policy PG estimation.

AES improves policy gradient performance by adaptively selecting experience.

Proposes a new policy gradient algorithm to improve reinforcement learning efficiency and stability.

Optimistic NPG improves policy optimization in online RL with efficient sample complexity.

Unified framework for understanding GRPO as U-statistic.

Softmax policy gradient achieves global optimality in wide neural networks with entropy regularization.

Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behavio…

We present a predictor-corrector framework, called PicCoLO, that can transform a first-order model-free reinforcement or imitation learning algorithm into a new hybrid method that leverages predictive models to accelerate policy learning. The new "PicCoLOed" algorithm optimizes a policy by recursively repeating two ste…

FPGs use structure to improve policy learning in complex tasks.

Direct optimization is an appealing framework that replaces integration with optimization of a random objective for approximating gradients in models with discrete random variables. A$^\star$ sampling is a framework for optimizing such random objectives over large spaces. We show how to combine these techniques to yiel…