arXiv research

Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…

STORM-PG uses momentum for faster policy gradient updates.

Study on policy gradient for stochastic bandits using diffusion approximation.

Large deviations theory applied to policy gradient methods.

Developed policy gradient methods for stochastic control with exit time, outperforming traditional techniques in share repurchase pricing.

Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stoc…

In this paper, we propose a novel reinforcement- learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation…

Develops DPG methods for continuous-time RL with deterministic policies.

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates across the action when estimating the gradient, instead of relying only on the action in the sampled trajectory. We es…

The paper analyzes the intrinsic exploration terms in policy-gradient algorithms.

Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining re…

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates (or sums) across actions when estimating the gradient, instead of relying only on the action in the sampled trajectory…

Short note on soft-max and policy gradients in bandit problems using Lyapunov functions.

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency.…

A method for accurate pricing of multidimensional derivatives under uncertain volatility.

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …

This paper investigates methods for estimating the optimal stochastic control policy for a Markov Decision Process with unknown transition dynamics and an unknown reward function. This form of model-free reinforcement learning comprises many real world systems such as playing video games, simulated control tasks, and r…

New method improves convergence of RL meta-learning.

New framework for policy gradient methods in continuous time reinforcement learning.

Policy gradient methods achieve linear convergence in simple MDPs.

ZDPG learns model-free policies without critics, improving on PG.

Memory-efficient algorithm reduces variance in off-policy RL.

Policy gradient methods converge for LQR problems with noisy state dynamics.

Improved sampling efficiency with Random Reshuffling for Langevin dynamics.

Study policy gradient and actor-critic methods for continuous-time reinforcement learning.

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …

Develops RL for dynamic risk assessment in stochastic optimization.

The paper interprets policy-gradient algorithms using continuation theory.

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment po…

Paper develops NPG for risk-averse RL with ECRMs, proving global convergence.

We study a reinforcement learning setting, where the state transition function is a convex combination of a stochastic continuous function and a deterministic function. Such a setting generalizes the widely-studied stochastic state transition setting, namely the setting of deterministic policy gradient (DPG). We firstl…

Policy gradients methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, even for simple control problems solvable by standard dynamic programming techniques, policy gradient algorithms face non-convex optimization pro…

Reinforcement learning is a promising approach to learning robotics controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomen…

Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.

Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies

Policy gradient method proves convergence in imperfect-information games.

DSPI connects natural policy gradient to policy iteration, proving global convergence.

This paper optimizes sampling policies for Bayesian optimization to improve exploration and exploitation.

We study a risk-constrained version of the stochastic shortest path (SSP) problem, where the risk measure considered is Conditional Value-at-Risk (CVaR). We propose two algorithms that obtain a locally risk-optimal policy by employing four tools: stochastic approximation, mini batches, policy gradients and importance s…

Paper presents a rank-1 approximation method for natural policy gradients in deep RL.

Study uses SGD to find near-optimal execution cost policies in dynamic markets.

Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.

Policy gradient methods find Nash equilibrium in noisy games.

This paper studies GAIL's global convergence for general MDP and nonlinear rewards.

Policy Gradient (PG) algorithms are among the best candidates for the much-anticipated applications of reinforcement learning to real-world control tasks, such as robotics. However, the trial-and-error nature of these methods poses safety issues whenever the learning process itself must be performed on a physical syste…

Stochastic variance-reduced gradient (SVRG) is an optimization method originally designed for tackling machine learning problems with a finite sum structure. SVRG was later shown to work for policy evaluation, a problem in reinforcement learning in which one aims to estimate the value function of a given policy. SVRG m…

A new method reduces variance in PG methods for RL, improving efficiency and convergence.