arXiv research

Actor-Critic method achieves optimal regret for unichain MDPs.

Study on natural actor-critic for POMDPs with finite memory.

This paper improves sample complexity for AC and NAC algorithms under Markovian sampling.

Paper provides convergence guarantees for off-policy NAC with finite sample complexity.

We describe a novel extension of soft actor-critics for hierarchical Deep Q-Networks (HDQN) architectures using mutual information metric. The proposed extension provides a suitable framework for encouraging explorations in such hierarchical networks. A natural utilization of this framework is an adversarial setting, w…

Optimistic actor-critic tackles linear MDPs with parametric policies.

Paper analyzes NAC with neural networks for efficient policy optimization.

Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…

The study uses AI to optimize trading in FX markets by considering size-dependent fees and risk-aversion.

This paper improves convergence bounds for AC and NAC algorithms with function approximation.

Improved exploration in RL with latent state marginalization.

Proposes three decentralized multi-agent reinforcement learning algorithms to reduce network congestion.

Paper analyzes convergence rates of two time-scale AC and NAC algorithms.

New algorithm for risk-sensitive reinforcement learning with natural policy gradients.

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

This work analyzes actor-critic methods for faster convergence.

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

Paper achieves $ε^{-2}$ sample complexity for actor-critic methods with minimal assumptions.

New PAC-Bayesian approach stabilizes actor-critic learning.

New reinforcement learning bound improves generalization for sequential data.

Multi-agent reinforcement learning has been successfully applied to a number of challenging problems. Despite these empirical successes, theoretical understanding of different algorithms is lacking, primarily due to the curse of dimensionality caused by the exponential growth of the state-action space with the number o…

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in…

New analysis shows actor-critic method converges efficiently in practical settings.

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…

WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.

Single-timescale actor-critic finds globally optimal policy.

A new method for risk-sensitive reinforcement learning using Spectral Risk Measures.

Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…

A new method improves actor-critic RL by integrating HMC, enhancing policy distribution and exploration.

TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution

After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the $L^2$ norm for the control variate estimators spanned by functions conditioned by the current state and acti…

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…

Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

Deep reinforcement learning boosts commodities trading performance.

Actor-critic algorithms converge to an ODE as data samples change dynamically.

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize…

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to dis…

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.

We develop an off-policy actor-critic algorithm for learning an optimal policy from a training set composed of data from multiple individuals. This algorithm is developed with a view towards its use in mobile health.

We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and R…

Study visualizes actor-critic loss landscapes for inventory optimization.

Generative Adversarial Networks (GANs) are a powerful framework for deep generative modeling. Posed as a two-player minimax problem, GANs are typically trained end-to-end on real-valued data and can be used to train a generator of high-dimensional and realistic images. However, a major limitation of GANs is that traini…

Paper introduces a meta-critic for accelerating off-policy actor-critic learning.