arXiv research

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…

New PAC-Bayesian approach stabilizes actor-critic learning.

Single-timescale actor-critic finds globally optimal policy.

Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize…

New analysis shows actor-critic method converges efficiently in practical settings.

This work analyzes actor-critic methods for faster convergence.

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

Paper introduces a meta-critic for accelerating off-policy actor-critic learning.

Actor-critic algorithms converge to an ODE as data samples change dynamically.

WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.

Neural networks trained with actor-critic algorithms converge to ODEs under weak convergence analysis.

Actor-Critic method achieves optimal regret for unichain MDPs.

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

AGAC uses an adversary to enhance exploration in complex tasks.

New method improves off-policy critic evaluation in reinforcement learning.

FORK improves model-free reinforcement learning performance.

TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution

Optimistic actor-critic tackles linear MDPs with parametric policies.

ESAC improves reinforcement learning by lookahead and intuition.

We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Differ…

MAGE optimizes policies using action gradients from model-based learning.

Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.

Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor). The update to the actor involves a log-likelihood update weighted by the action-values, with the addition of entropy regularization for soft variants. In th…

A new method improves actor-critic RL by integrating HMC, enhancing policy distribution and exploration.

New testing method for robust actor-critic bandit algorithms.

After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the $L^2$ norm for the control variate estimators spanned by functions conditioned by the current state and acti…

Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

This paper improves convergence bounds for AC and NAC algorithms with function approximation.

USAC balances pessimism and optimism in actor-critic training for better exploration and performance.

PEARL uses reinforcement learning to improve matrix preconditioners.

New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.

This work analyzes how neural networks learn representations in actor-critic algorithms.

Paper analyzes convergence rates of two time-scale AC and NAC algorithms.

Study visualizes actor-critic loss landscapes for inventory optimization.

Improved sample complexity for actor-critic algorithms in MDPs.

Improved convergence for actor-critic algorithms in MDPs.

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yieldi…

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to dis…

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at …

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…

Proposes a value-based method for continuous control without an actor.

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

New actor-critic algorithm achieves optimal sample efficiency in RL.