arXiv research

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency.…

Study policy gradient and actor-critic methods for continuous-time reinforcement learning.

Develops first-order methods for average-reward MDPs with strong guarantees.

This work analyzes actor-critic methods for faster convergence.

Study on sample complexity of policy gradient for stabilizing linear systems under multiplicative noise.

This work overcomes bias in concave multi-objective reinforcement learning.

A new method solves bilevel optimization problems in competitive Markov games.

We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of stochastic policy gradient for the realized MDP. In particular, using stoc…

ZOSPI improves RL policies with global value function exploitation.

STORM-PG uses momentum for faster policy gradient updates.

DBQPG improves policy gradient estimation with fewer samples.

Continuous-time Q-learning theory developed for reinforcement learning.

Policy gradient method proves convergence in imperfect-information games.

Large deviations theory applied to policy gradient methods.

The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The …

Short note on soft-max and policy gradients in bandit problems using Lyapunov functions.

Optimizes Thompson sampling policies using policy gradient methods.

WMPG reduces policy gradient variance using world models.

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates across the action when estimating the gradient, instead of relying only on the action in the sampled trajectory. We es…

New $k$-step policy gradient method avoids local optima in restricted policy classes.

Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of action…

Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.

Policy gradient methods achieve linear convergence in simple MDPs.

Reinforcement learning is a promising approach to learning robotics controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomen…

Paper proposes a policy gradient method for confounded POMDPs.

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …

Study on policy gradient for stochastic bandits using diffusion approximation.

Policy gradient algorithm with variable learning rates achieves near-optimal performance in multi-arm bandit problems.

Policy gradient methods find Nash equilibrium in noisy games.

New method QMLE performs well in complex action spaces without policy gradients.

The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge in this scenario is how to reduce the variance of policy gradient estimates for reliable policy updates. In this paper, we combine the followi…

Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at …

We study a reinforcement learning setting, where the state transition function is a convex combination of a stochastic continuous function and a deterministic function. Such a setting generalizes the widely-studied stochastic state transition setting, namely the setting of deterministic policy gradient (DPG). We firstl…

Develops DPG methods for continuous-time RL with deterministic policies.

Softmax policy gradient methods converge at $O(1/t)$ rate with constants depending on problem and initialization.

We study a policy gradient method with L2 regularization for MAB problems.

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…

Policy-gradient method controls multiple non-cohesive targets.

New method for RL with general utilities using variational policy gradient.

New framework for policy gradient methods in continuous time reinforcement learning.

Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effe…

Inspired by the seminal work on Stein Variational Inference and Stein Variational Policy Gradient, we derived a method to generate samples from the posterior variational parameter distribution by \textit{explicitly} minimizing the KL divergence to match the target distribution in an amortize fashion. Consequently, we a…

Paper presents a new policy gradient theorem using weak derivatives for reinforcement learning.

New method speeds up lifelong learning of complex tasks.

Policy gradient is a generic and flexible reinforcement learning approach that generally enjoys simplicity in analysis, implementation, and deployment. In the last few decades, this approach has been extensively advanced for fully observable environments. In this paper, we generalize a variety of these advances to part…

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates (or sums) across actions when estimating the gradient, instead of relying only on the action in the sampled trajectory…

Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stoc…

FACMAC combines deep policy gradients with factored critic for multi-agent reinforcement learning.