This work analyzes actor-critic methods for faster convergence.
problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.
method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.
result Actor-critic method finds a first-order stationary point with i l d e O ( ε − 2.5 ) \mathcal{ ilde{O}}(ε^{-2.5}) i l d e O ( ε − 2.5 ) sample complexity. Improves stability of actor-critic methods by penalizing TD error.
problem Instability in actor-critic methods during learning.
method Regularize actor's learning objective by penalizing critic's TD error.
result Improves stability and overall performance of actor-critic methods.
Improved A2C method with lower variance.
problem Reducing variance in deep policy gradient methods.
method Using control variate theory, derived a new A2C formulation with lower variance.
result New A2C method has lower variance and improved performance.
The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.
problem Lack of theoretical understanding of target networks in actor-critic methods.
method Proposes a theoretical analysis of an online target-based actor-critic algorithm with linear function approximation.
result Establishes asymptotic convergence results and finite-time analysis for both critic and actor.
Paper proves convergence of actor-critic methods for MDPs.
problem Finding approximate solutions to entropy-regularized MDPs.
method Stochastic approximation and policy evaluation with general regularizers.
result Probability one convergence of actor-critic algorithms.
New analysis shows actor-critic method converges efficiently in practical settings.
problem Understanding finite-time convergence of single-timescale actor-critic methods.
method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.
result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).
Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…
A new method improves actor-critic RL by integrating HMC, enhancing policy distribution and exploration.
problem Actor-critic RL yields suboptimal policies due to amortization gap and insufficient exploration.
method Integrating Hamiltonian Monte Carlo (HMC) into the actor-critic RL framework.
result Improves policy distribution and exploration, leading to better policy estimates and higher returns.
WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.
problem Inherent instability in actor-critic reinforcement learning algorithms.
method Wasserstein adaptive value estimation with Sinkhorn approximation.
result Achieves $\mathcal{O}\left(\frac{1}{k}
ight)$ convergence rate for critic's mean squared error.
Optimistic actor-critic tackles linear MDPs with parametric policies.
problem Theoretical limitations of existing actor-critic methods for linear MDPs.
method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.
result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.
Actor-Critic method achieves optimal regret for unichain MDPs.
problem Scalable regret analysis for infinite-horizon average-reward MDPs.
method NAC-B, a Natural Actor-Critic with batching.
result Order-optimal regret of i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) in infinite-horizon average-reward MDPs. Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…
Hybrid actor-critic learns in complex action spaces.
problem Learning in complex, structured action spaces.
method Parallel sub-actor networks and a critic network.
result Hybrid PPO outperforms previous methods in parameterized action spaces.
DAC combines actor-critic methods for learning options.
problem Learning hierarchical policies in reinforcement learning.
method Double Actor-Critic (DAC) architecture using two parallel augmented MDPs.
result DAC outperforms existing methods in robot simulation tasks.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
Distributed learning method for multi-agent reinforcement learning with policy coordination.
problem Solving multi-agent reinforcement learning problems with coordination.
method Distributed off-policy actor critic with policy consensus.
result The proposed algorithm achieves asymptotic agreement on the global optimal policy function.
Efficient actor-critic learning with shared experience replay improves data efficiency.
problem Challenges in actor-critic reinforcement learning with experience replay and off-policy learning stability.
method Combining actor-critic algorithms with shared experience replay, analyzing V-trace, proposing a trust region scheme.
result State-of-the-art data efficiency on Atari achieved with 200M environment frames.
New actor-critic method reduces sample complexity for reinforcement learning.
problem Improving sample complexity for actor-critic algorithms in reinforcement learning.
method Integrates Monte Carlo rollouts into policy search steps for better control over bias.
result Established sample complexity for actor-critic algorithms with policy gradient.
Study visualizes actor-critic loss landscapes for inventory optimization.
problem Difficulties in solving multi-store dynamic inventory control problems.
method Low-dimensional visualizations of actor loss function.
result Loss landscapes favor optimal policies in reinforcement learning.
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
problem Lack of precise theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
method Adapting a finite-time convergence analysis to the asymmetric actor-critic setting with linear function approximators.
result A finite-time bound reveals that the asymmetric critic eliminates aliasing errors in the agent state.
Actor-critic converges globally in LQR with ergodic cost.
problem Theoretical understanding of actor-critic algorithm's global convergence.
method Nonasymptotic convergence analysis of actor-critic in linear quadratic regulator (LQR) setting.
result Actor-critic finds globally optimal policy and value function at a linear rate.
Paper introduces a meta-critic for accelerating off-policy actor-critic learning.
problem Improving sample efficiency in continuous control tasks.
method Meta-critic that meta-learns an additional loss for the actor.
result Online meta-critic learning leads to improved performance in various continuous control environments.
Optimistic Actor-Critic improves exploration efficiency in reinforcement learning.
problem Poor sample efficiency in existing actor-critic methods.
method Introduces Optimistic Actor-Critic, approximating upper and lower bounds on state-action value function.
result Achieves state-of-the-art sample efficiency in challenging continuous control tasks.
Study on natural actor-critic for POMDPs with finite memory.
problem Learning in partially observed Markov decision processes with noisy observations.
method Finite actor-critic method with multi-step temporal difference learning.
result First non-asymptotic global convergence for POMDPs with function approximation.
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…
New Soft Actor-Critic for discrete actions.
problem Applying reinforcement learning to games with discrete actions.
method Derived an alternative Soft Actor-Critic for discrete actions.
result Competitive with tuned model-free state-of-the-art on Atari games.
Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.
problem Continuous-time mean-field control in reinforcement learning.
method Actor-critic approach with entropy regularisation, value function alternation, and Wasserstein space parametrisation.
result Derives exact parametrisation of actor and critic functions in linear-quadratic mean-field framework.
Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.
problem Performance degradation and overfitting in actor-critic models with smaller actors.
method Broad empirical investigations and analyses of asymmetric actor-critic setups, exploring techniques to mitigate value underestimation.
result Value underestimation is a key cause of performance degradation in smaller actor-critic models, and the critic plays a crucial role in mitigating this.
New method improves deep policy gradient algorithms by learning relative state values.
problem High sample complexity and instability in policy gradient methods.
method Uses a new state-value function approximation based on residual variance.
result Empirical improvement across diverse continuous control tasks and algorithms.
Paper develops a new multi-agent reinforcement learning algorithm.
problem Improving policies in a network of communicating agents.
method Develops a multi-agent off-policy actor-critic algorithm using emphatic temporal difference learning.
result Proves convergence of the algorithm under linear function approximation.
New PAC-Bayesian approach stabilizes actor-critic learning.
problem Training instability in actor-critic algorithms.
method Employing PAC-Bayesian bound as the critic training objective.
result Significant improvement in online learning performance.
AACC improves RL performance in changing environments.
problem Deterioration of RL performance in real-world tasks.
method Formalizes CMDPs, proposes AACC for contextual RL.
result AACC outperforms baselines in various simulated environments.
PEARL uses reinforcement learning to improve matrix preconditioners.
problem Learning effective preconditioners for iterative solvers is challenging.
method PEARL employs an actor-critic reinforcement learning framework to learn preconditioners dynamically.
result PEARL outperforms traditional and neural preconditioners in flexibility and solving speed.
SAC improves deep RL by balancing reward and randomness.
problem High sample complexity and hyperparameter brittleness in RL.
method Maximum entropy framework, constrained optimization, temperature tuning.
result SAC achieves state-of-the-art performance and stability.
COF-PAC converges with novel critic and learning method.
problem Convergent off-policy actor-critic with function approximation.
method Two-timescale approach with Gradient Emphasis Learning (GEM).
result First provably convergent COF-PAC with linear critics and nonlinear actor.
AGAC uses an adversary to enhance exploration in complex tasks.
problem Sample inefficiency in complex environments, especially in tasks requiring efficient exploration.
method Integrates an adversary into the actor-critic framework to encourage innovative exploration strategies.
result AGAC leads to more exhaustive exploration and outperforms state-of-the-art methods.
New algorithms improve robust reinforcement learning under uncertainty.
problem Robust reinforcement learning in MDPs with contamination.
method Non-asymptotic convergence analysis of Q Q Q -learning and actor-critic methods. result Efficient algorithms learn robust policies with minimal samples.
A new method for continuous control avoids local movement issues.
problem Limitations of policy gradient methods in continuous control.
method Distributional framework and Generative Actor Critic (GAC) method.
result GAC outperforms policy gradient methods in continuous domains.
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.
problem Solving mean-field control problems in continuous time reinforcement learning.
method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.
result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.
Both generative adversarial networks (GAN) in unsupervised learning and actor-critic methods in reinforcement learning (RL) have gained a reputation for being difficult to optimize. Practitioners in both fields have amassed a large number of strategies to mitigate these instabilities and improve training. Here we show …
We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…
Single-timescale actor-critic finds globally optimal policy.
problem Finding globally optimal policy in reinforcement learning.
method Simultaneous actor and critic updates with linear or deep neural network approximations.
result Actor sequence converges to globally optimal policy at O ( K − 1 / 2 ) O(K^{-1/2}) O ( K − 1/2 ) rate. Enhances SAC for better sample efficiency in continuous-action tasks.
problem Improving sample efficiency in soft actor-critic algorithms.
method Integrating Emphasizing Recent Experience (ERE) with Soft Actor-Critic (SAC) and Priority Experience Replay (PER).
result ERE significantly improves sample efficiency compared to vanilla SAC, especially for continuous-action tasks.
TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution
problem Optimal execution of large stock sell programs
method Twin-Target Deterministic Actor-Critic with Policy Smoothing
result Reduces mean implementation shortfall percentage
Enhances reinforcement learning with partial state information.
problem Improving learning under partial observability with limited privileged signals.
method Introduced informed asymmetric actor-critic framework that uses arbitrary state-dependent privileged signals.
result Unbiased policy gradient estimates with arbitrary privileged signals.