arXiv research

This work analyzes actor-critic methods for faster convergence.

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the $L^2$ norm for the control variate estimators spanned by functions conditioned by the current state and acti…

New analysis shows actor-critic method converges efficiently in practical settings.

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…

A new method improves actor-critic RL by integrating HMC, enhancing policy distribution and exploration.

WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize…

Optimistic actor-critic tackles linear MDPs with parametric policies.

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

Actor-Critic method achieves optimal regret for unichain MDPs.

Study visualizes actor-critic loss landscapes for inventory optimization.

A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.

Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

Paper introduces a meta-critic for accelerating off-policy actor-critic learning.

Study on natural actor-critic for POMDPs with finite memory.

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.

Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.

In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional…

New method improves deep policy gradient algorithms by learning relative state values.

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

AACC improves RL performance in changing environments.

New PAC-Bayesian approach stabilizes actor-critic learning.

PEARL uses reinforcement learning to improve matrix preconditioners.

AGAC uses an adversary to enhance exploration in complex tasks.

We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ …

New algorithms improve robust reinforcement learning under uncertainty.

USAC balances pessimism and optimism in actor-critic training for better exploration and performance.

In this paper, we present a probability one convergence proof, under suitable conditions, of a certain class of actor-critic algorithms for finding approximate solutions to entropy-regularized MDPs using the machinery of stochastic approximation. To obtain this overall result, we prove the convergence of policy evaluat…

New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.

Both generative adversarial networks (GAN) in unsupervised learning and actor-critic methods in reinforcement learning (RL) have gained a reputation for being difficult to optimize. Practitioners in both fields have amassed a large number of strategies to mitigate these instabilities and improve training. Here we show …

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…

Single-timescale actor-critic finds globally optimal policy.

Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that…

TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution

This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-…

Enhances reinforcement learning with partial state information.

New actor-critic algorithm achieves optimal sample efficiency in RL.

Improved exploration in RL with latent state marginalization.

Paper achieves $ε^{-2}$ sample complexity for actor-critic methods with minimal assumptions.

New testing method for robust actor-critic bandit algorithms.

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address th…

AEA dynamically aggregates ensemble targets for actor-critic learning.

Actor-critic algorithms converge to an ODE as data samples change dynamically.