The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.
problem Lack of theoretical understanding of target networks in actor-critic methods.
method Proposes a theoretical analysis of an online target-based actor-critic algorithm with linear function approximation.
result Establishes asymptotic convergence results and finite-time analysis for both critic and actor.
New Soft Actor-Critic for discrete actions.
problem Applying reinforcement learning to games with discrete actions.
method Derived an alternative Soft Actor-Critic for discrete actions.
result Competitive with tuned model-free state-of-the-art on Atari games.
Actor-critic converges globally in LQR with ergodic cost.
problem Theoretical understanding of actor-critic algorithm's global convergence.
method Nonasymptotic convergence analysis of actor-critic in linear quadratic regulator (LQR) setting.
result Actor-critic finds globally optimal policy and value function at a linear rate.
New PAC-Bayesian approach stabilizes actor-critic learning.
problem Training instability in actor-critic algorithms.
method Employing PAC-Bayesian bound as the critic training objective.
result Significant improvement in online learning performance.
We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…
DAC combines actor-critic methods for learning options.
problem Learning hierarchical policies in reinforcement learning.
method Double Actor-Critic (DAC) architecture using two parallel augmented MDPs.
result DAC outperforms existing methods in robot simulation tasks.
Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
problem Lack of precise theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
method Adapting a finite-time convergence analysis to the asymmetric actor-critic setting with linear function approximators.
result A finite-time bound reveals that the asymmetric critic eliminates aliasing errors in the agent state.
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
Actor-critic algorithms converge to an ODE as data samples change dynamically.
problem Challenging to mathematically analyze due to non-i.i.d. data samples.
method Proved convergence to an ODE using time rescaling and geometric ergodicity.
result Convergence to the ODE limit and its properties proven.
New method improves off-policy critic evaluation in reinforcement learning.
problem High variance and instability in off-policy policy evaluation.
method Doubly robust estimators applied to actor-critic algorithms.
result Doubly robust estimation significantly improves performance in continuous control tasks.
We develop an off-policy actor-critic algorithm for learning an optimal policy from a training set composed of data from multiple individuals. This algorithm is developed with a view towards its use in mobile health.
Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…
New analysis shows actor-critic method converges efficiently in practical settings.
problem Understanding finite-time convergence of single-timescale actor-critic methods.
method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.
result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).
We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …
Paper proves convergence of actor-critic methods for MDPs.
problem Finding approximate solutions to entropy-regularized MDPs.
method Stochastic approximation and policy evaluation with general regularizers.
result Probability one convergence of actor-critic algorithms.
TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution
problem Optimal execution of large stock sell programs
method Twin-Target Deterministic Actor-Critic with Policy Smoothing
result Reduces mean implementation shortfall percentage
Optimistic actor-critic tackles linear MDPs with parametric policies.
problem Theoretical limitations of existing actor-critic methods for linear MDPs.
method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.
result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.
We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and R…
New algorithm for multi-agent reinforcement learning with reduced communication.
problem Cooperative learning among multiple agents with limited communication.
method Randomized multi-agent actor-critic algorithm for directed graphs.
result Algorithm solves problem for strongly connected graphs with reduced communication.
Distributed learning method for multi-agent reinforcement learning with policy coordination.
problem Solving multi-agent reinforcement learning problems with coordination.
method Distributed off-policy actor critic with policy consensus.
result The proposed algorithm achieves asymptotic agreement on the global optimal policy function.
This work analyzes actor-critic methods for faster convergence.
problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.
method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.
result Actor-critic method finds a first-order stationary point with i l d e O ( ε − 2.5 ) \mathcal{ ilde{O}}(ε^{-2.5}) i l d e O ( ε − 2.5 ) sample complexity. FORK improves model-free reinforcement learning performance.
problem Improving model-free reinforcement learning performance.
method Introducing a new forward-looking Actor (FORK) for Actor-Critic algorithms.
result FORK significantly improves performance in various environments.
Paper develops a new multi-agent reinforcement learning algorithm.
problem Improving policies in a network of communicating agents.
method Develops a multi-agent off-policy actor-critic algorithm using emphatic temporal difference learning.
result Proves convergence of the algorithm under linear function approximation.
New algorithms improve robust reinforcement learning under uncertainty.
problem Robust reinforcement learning in MDPs with contamination.
method Non-asymptotic convergence analysis of Q Q Q -learning and actor-critic methods. result Efficient algorithms learn robust policies with minimal samples.
New testing method for robust actor-critic bandit algorithms.
problem Balancing data collection for app performance and user adherence.
method Modified actor-critic algorithm and novel testing procedure.
result Testing procedure is robust to critic misspecification.
Optimistic Actor-Critic improves exploration efficiency in reinforcement learning.
problem Poor sample efficiency in existing actor-critic methods.
method Introduces Optimistic Actor-Critic, approximating upper and lower bounds on state-action value function.
result Achieves state-of-the-art sample efficiency in challenging continuous control tasks.
Hybrid actor-critic learns in complex action spaces.
problem Learning in complex, structured action spaces.
method Parallel sub-actor networks and a critic network.
result Hybrid PPO outperforms previous methods in parameterized action spaces.
WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.
problem Inherent instability in actor-critic reinforcement learning algorithms.
method Wasserstein adaptive value estimation with Sinkhorn approximation.
result Achieves $\mathcal{O}\left(\frac{1}{k}
ight)$ convergence rate for critic's mean squared error.
Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.
problem Continuous-time mean-field control in reinforcement learning.
method Actor-critic approach with entropy regularisation, value function alternation, and Wasserstein space parametrisation.
result Derives exact parametrisation of actor and critic functions in linear-quadratic mean-field framework.
Neural networks trained with actor-critic algorithms converge to ODEs under weak convergence analysis.
problem Challenges in convergence analysis due to changing data distributions in online learning.
method Geometric ergodicity of data samples, Poisson equation, weak convergence techniques.
result Actor and critic networks converge to solutions of ODEs with random initial conditions.
Improved convergence for actor-critic algorithms in MDPs.
problem Global convergence analysis for actor-critic algorithms in MDPs.
method Introduced an analytical framework to handle complex recursions, established convergence to ε-close globally optimal policy with improved sample complexity.
result Converges to ε-close globally optimal policy with sample complexity of O(ε^(-3)) compared to O(ε^(-2)) for ε-close stationary policy.
New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.
problem Solving mean-field control problems in continuous time reinforcement learning.
method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.
result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.
New actor-critic algorithm achieves optimal sample efficiency in RL.
problem Achieving ε ε ε -optimal policies with minimal samples in RL. method Integrates optimism, off-policy critic estimation, and rare-switching policy resets.
result Sample complexity of O ( d H 5 log ∣ A ∣ / ε 2 + d H 4 log ∣ F ∣ / ε 2 ) O(dH^5 \log|\mathcal{A}|/ε^2 + d H^4 \log|\mathcal{F}|/ ε^2) O ( d H 5 log ∣ A ∣/ ε 2 + d H 4 log ∣ F ∣/ ε 2 ) trajectories. Actor-critic finds Nash in mean-field games with linear-quadratic costs.
problem Finding Nash equilibrium in mean-field Markov games with linear dynamics and quadratic costs.
method Mean-field actor-critic algorithm with linear function approximation.
result Algorithm converges linearly to Nash equilibrium.
Improved sample complexity for actor-critic algorithms in MDPs.
problem Achieving optimal policies with limited data in reinforcement learning.
method Single-timescale actor-critic with STORM (STOchastic Recursive Momentum) and a sample buffer.
result Optimal sample complexity of O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) for ε ε ε -optimal policies. Self-Tuning Actor-Critic improves reinforcement learning performance.
problem Manual hyperparameter tuning is time-consuming and domain-specific.
method Uses metagradients for online hyperparameter adaptation.
result Improves performance across various domains and tasks.
New RL framework improves real-time control performance.
problem Real-time RL systems assume static states, leading to suboptimal outcomes.
method Introduces a new real-time RL framework where states and actions evolve simultaneously.
result RTAC algorithm outperforms existing state-of-the-art algorithms in real-time and non-real-time settings.
Efficient actor-critic learning with shared experience replay improves data efficiency.
problem Challenges in actor-critic reinforcement learning with experience replay and off-policy learning stability.
method Combining actor-critic algorithms with shared experience replay, analyzing V-trace, proposing a trust region scheme.
result State-of-the-art data efficiency on Atari achieved with 200M environment frames.
Global convergence proved for multi-agent LQRs with hierarchical actor-critic.
problem Challenges in understanding multi-agent reinforcement learning algorithms.
method Developed a hierarchical actor-critic algorithm for partially exchangeable agents.
result Global linear convergence to optimal policy proved.
Actor-Critic method achieves optimal regret for unichain MDPs.
problem Scalable regret analysis for infinite-horizon average-reward MDPs.
method NAC-B, a Natural Actor-Critic with batching.
result Order-optimal regret of i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) in infinite-horizon average-reward MDPs. New method improves deep policy gradient algorithms by learning relative state values.
problem High sample complexity and instability in policy gradient methods.
method Uses a new state-value function approximation based on residual variance.
result Empirical improvement across diverse continuous control tasks and algorithms.
Deep learning model solves high-dimensional PDEs using Actor-Critic approach.
problem Solving high-dimensional nonlinear PDEs efficiently.
method Reformulated PDE into BSDE system, inspired by Actor-Critic algorithm for deep RL.
result Improved model with fewer parameters, faster convergence, and less hyperparameter tuning.
Improved exploration in SAC using Normalizing Flows policies.
problem Brittleness and inefficiency of DRL algorithms in continuous action spaces.
method Introducing Normalizing Flow policies within the SAC framework to learn more expressive policies.
result Increased stability and better exploration in sparse reward settings.
Deep actor-critic learning optimizes power control in mobile networks.
problem Optimizing power control in large-scale wireless mobile networks.
method Multi-agent deep reinforcement learning with deep deterministic policy gradient.
result The algorithm maximizes a global utility function in a distributed manner.
New actor-critic method reduces sample complexity for reinforcement learning.
problem Improving sample complexity for actor-critic algorithms in reinforcement learning.
method Integrates Monte Carlo rollouts into policy search steps for better control over bias.
result Established sample complexity for actor-critic algorithms with policy gradient.
Single-timescale actor-critic finds globally optimal policy.
problem Finding globally optimal policy in reinforcement learning.
method Simultaneous actor and critic updates with linear or deep neural network approximations.
result Actor sequence converges to globally optimal policy at O ( K − 1 / 2 ) O(K^{-1/2}) O ( K − 1/2 ) rate. Study efficient convergence of RL algorithm with function approximation.
problem Convergence of actor-critic algorithm with nonlinear function approximation.
method Stochastic gradient descent ascent with adaptive proximal term, Polyak-Łojasiewicz condition.
result First efficient convergence result with rate of O(sqrt{ln(N d G^2) / N}).
Enhances SAC for better sample efficiency in continuous-action tasks.
problem Improving sample efficiency in soft actor-critic algorithms.
method Integrating Emphasizing Recent Experience (ERE) with Soft Actor-Critic (SAC) and Priority Experience Replay (PER).
result ERE significantly improves sample efficiency compared to vanilla SAC, especially for continuous-action tasks.