On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “actor-critic algorithms”

Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcs.LG

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

problem Lack of theoretical understanding of target networks in actor-critic methods.

method Proposes a theoretical analysis of an online target-based actor-critic algorithm with linear function approximation.

result Establishes asymptotic convergence results and finite-time analysis for both critic and actor.

Actor-critic Target networks Deep reinforcement learning Temporal difference learning Markovian sampling

2021-06-14abs ↗pdf ↗

PAC-Bayesian Soft Actor-Critic Learningcs.LG

New PAC-Bayesian approach stabilizes actor-critic learning.

problem Training instability in actor-critic algorithms.

method Employing PAC-Bayesian bound as the critic training objective.

result Significant improvement in online learning performance.

actor-critic reinforcement learning PAC-Bayesian Soft Actor-Critic stability

2023-01-30abs ↗pdf ↗

Variance Adjusted Actor Critic Algorithmsstat.ML

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…

2013-10-14abs ↗pdf ↗

A Theoretical Justification for Asymmetric Actor-Critic Algorithmscs.LG

Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

problem Lack of precise theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

method Adapting a finite-time convergence analysis to the asymmetric actor-critic setting with linear function approximators.

result A finite-time bound reveals that the asymmetric critic eliminates aliasing errors in the agent state.

reinforcement learning asymmetric learning finite-time convergence actor-critic algorithms linear function approximators

2025-01-31abs ↗pdf ↗

On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Costcs.LG

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

2019-07-14abs ↗pdf ↗

An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processescs.LG

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

problem Risk-sensitive cost criterion in Markov decision processes.

method Actor-critic algorithm with function approximation.

result Asymptotic convergence of the actor-critic algorithm.

actor-critic Markov decision processes risk-sensitive function approximation asymptotic convergence

2025-02-17abs ↗pdf ↗

Global Convergence of the ODE Limit for Online Actor-Critic Algorithms in Reinforcement Learningcs.LG

Actor-critic algorithms converge to an ODE as data samples change dynamically.

problem Challenging to mathematically analyze due to non-i.i.d. data samples.

method Proved convergence to an ODE using time rescaling and geometric ergodicity.

result Convergence to the ODE limit and its properties proven.

Actor-Critic Reinforcement Learning Ordinary Differential Equation Geometric Ergodicity Convergence Rate

2021-08-19abs ↗pdf ↗

A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Rewardstat.ML

We develop an off-policy actor-critic algorithm for learning an optimal policy from a training set composed of data from multiple individuals. This algorithm is developed with a view towards its use in mobile health.

2016-07-18abs ↗pdf ↗

Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscs.AI

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

2018-01-31abs ↗pdf ↗

Finite-time analysis of single-timescale actor-criticcs.LG

New analysis shows actor-critic method converges efficiently in practical settings.

problem Understanding finite-time convergence of single-timescale actor-critic methods.

method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.

result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).

actor-critic finite-time analysis single-timescale reinforcement learning Markovian sampling

2022-10-18abs ↗pdf ↗

Mean Actor Criticstat.ML

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

2017-09-01abs ↗pdf ↗

TT-DAC-PS: Twin-Target Deterministic Actor-Critic with Policy Smoothing for Optimal Trade Executioncs.AI

TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution

problem Optimal execution of large stock sell programs

method Twin-Target Deterministic Actor-Critic with Policy Smoothing

result Reduces mean implementation shortfall percentage

trade execution stock sell programs reinforcement learning optimization

2026-06-07abs ↗pdf ↗

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learningcs.LG

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …

2019-12-11abs ↗pdf ↗

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processescs.LG

Optimistic actor-critic tackles linear MDPs with parametric policies.

problem Theoretical limitations of existing actor-critic methods for linear MDPs.

method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.

result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.

actor-critic linear MDPs parametric policies logit-matching Thompson sampling

2026-03-30abs ↗pdf ↗

Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learningcs.LG

We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and R…

2019-02-14abs ↗pdf ↗

Soft Actor-Critic for Discrete Action Settingscs.LG

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to dis…

2019-10-16abs ↗pdf ↗

A Finite Time Analysis of Two Time-Scale Actor Critic Methodscs.LG

This work analyzes actor-critic methods for faster convergence.

problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.

method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.

result Actor-critic method finds a first-order stationary point with

\mathcal{ ilde{O}}(ε^{-2.5})

sample complexity.

actor-critic two-time-scale finite-time analysis sample complexity reinforcement learning

2020-05-04abs ↗pdf ↗

FORK: A Forward-Looking Actor For Model-Free Reinforcement Learningcs.LG

FORK improves model-free reinforcement learning performance.

problem Improving model-free reinforcement learning performance.

method Introducing a new forward-looking Actor (FORK) for Actor-Critic algorithms.

result FORK significantly improves performance in various environments.

reinforcement learning actor-critic model-free performance improvement forward-looking

2020-10-04abs ↗pdf ↗

Efficient

Q

-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learningcs.LG

New algorithms improve robust reinforcement learning under uncertainty.

problem Robust reinforcement learning in MDPs with contamination.

method Non-asymptotic convergence analysis of

Q

-learning and actor-critic methods.

result Efficient algorithms learn robust policies with minimal samples.

reinforcement learning robustness Q-learning actor-critic Markov decision processes

2025-06-08abs ↗pdf ↗

Robust Tests in Online Decision-Makingstat.ML

New testing method for robust actor-critic bandit algorithms.

problem Balancing data collection for app performance and user adherence.

method Modified actor-critic algorithm and novel testing procedure.

result Testing procedure is robust to critic misspecification.

bandit algorithms actor-critic mobile health data collection app performance

2022-08-21abs ↗pdf ↗

Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learningcs.LG

WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.

problem Inherent instability in actor-critic reinforcement learning algorithms.

method Wasserstein adaptive value estimation with Sinkhorn approximation.

result Achieves $\mathcal{O}\left(\frac{1}{k} ight)$ convergence rate for critic's mean squared error.

reinforcement learning actor-critic Wasserstein stability Sinkhorn

2025-01-17abs ↗pdf ↗

Actor-Critic learning for mean-field control in continuous timestat.ML

Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.

problem Continuous-time mean-field control in reinforcement learning.

method Actor-critic approach with entropy regularisation, value function alternation, and Wasserstein space parametrisation.

result Derives exact parametrisation of actor and critic functions in linear-quadratic mean-field framework.

actor-critic mean-field control reinforcement learning entropy regularisation Wasserstein space

2023-03-13abs ↗pdf ↗

DAC: The Double Actor-Critic Architecture for Learning Optionscs.LG

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yieldi…

2019-04-29abs ↗pdf ↗

On the Convergence of Single-Timescale Actor-Criticcs.LG

Improved convergence for actor-critic algorithms in MDPs.

problem Global convergence analysis for actor-critic algorithms in MDPs.

method Introduced an analytical framework to handle complex recursions, established convergence to ε-close globally optimal policy with improved sample complexity.

result Converges to ε-close globally optimal policy with sample complexity of O(ε^(-3)) compared to O(ε^(-2)) for ε-close stationary policy.

actor-critic Markov Decision Processes convergence global optimality sample complexity

2024-10-11abs ↗pdf ↗

Weak Convergence Analysis of Online Neural Actor-Critic Algorithmscs.LG

Neural networks trained with actor-critic algorithms converge to ODEs under weak convergence analysis.

problem Challenges in convergence analysis due to changing data distributions in online learning.

method Geometric ergodicity of data samples, Poisson equation, weak convergence techniques.

result Actor and critic networks converge to solutions of ODEs with random initial conditions.

Neural networks Actor-critic Convergence analysis Weak convergence Geometric ergodicity

2024-03-25abs ↗pdf ↗

Actor critic learning algorithms for mean-field control with moment neural networksstat.ML

New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.

problem Solving mean-field control problems in continuous time reinforcement learning.

method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.

result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.

mean-field control actor-critic moment neural networks Wasserstein space reinforcement learning

2023-09-08abs ↗pdf ↗

Actor-Critics Can Achieve Optimal Sample Efficiencystat.ML

New actor-critic algorithm achieves optimal sample efficiency in RL.

problem Achieving

ε

-optimal policies with minimal samples in RL.

method Integrates optimism, off-policy critic estimation, and rare-switching policy resets.

result Sample complexity of

O(dH^5 \log|\mathcal{A}|/ε^2 + d H^4 \log|\mathcal{F}|/ ε^2)

trajectories.

Actor-Critic Reinforcement Learning Optimism Off-Policy Rare-Switching

2025-05-06abs ↗pdf ↗

Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentumcs.LG

Improved sample complexity for actor-critic algorithms in MDPs.

problem Achieving optimal policies with limited data in reinforcement learning.

method Single-timescale actor-critic with STORM (STOchastic Recursive Momentum) and a sample buffer.

result Optimal sample complexity of

O(ε^{-2})

for

ε

-optimal policies.

actor-critic reinforcement learning sample complexity Markov decision processes Momentum

2026-02-02abs ↗pdf ↗

A Self-Tuning Actor-Critic Algorithmstat.ML

Self-Tuning Actor-Critic improves reinforcement learning performance.

problem Manual hyperparameter tuning is time-consuming and domain-specific.

method Uses metagradients for online hyperparameter adaptation.

result Improves performance across various domains and tasks.

Reinforcement Learning Hyperparameter Tuning Actor-Critic Metagradients Self-Tuning

2020-02-28abs ↗pdf ↗

Regret Analysis of Average-Reward Unichain MDPs via an Actor-Critic Approachcs.LG

Actor-Critic method achieves optimal regret for unichain MDPs.

problem Scalable regret analysis for infinite-horizon average-reward MDPs.

method NAC-B, a Natural Actor-Critic with batching.

result Order-optimal regret of

ilde{O}(\sqrt{T})

in infinite-horizon average-reward MDPs.

Actor-Critic Natural Actor-Critic MDP Regret Analysis Unichain

2025-05-26abs ↗pdf ↗

Learning Value Functions in Deep Policy Gradients using Residual Variancecs.LG

New method improves deep policy gradient algorithms by learning relative state values.

problem High sample complexity and instability in policy gradient methods.

method Uses a new state-value function approximation based on residual variance.

result Empirical improvement across diverse continuous control tasks and algorithms.

policy gradients deep learning value functions residual variance actor-critic

2020-10-09abs ↗pdf ↗

Actor-Critic Algorithm for High-dimensional Partial Differential Equationscs.LG

Deep learning model solves high-dimensional PDEs using Actor-Critic approach.

problem Solving high-dimensional nonlinear PDEs efficiently.

method Reformulated PDE into BSDE system, inspired by Actor-Critic algorithm for deep RL.

result Improved model with fewer parameters, faster convergence, and less hyperparameter tuning.

Deep Learning Partial Differential Equations Actor-Critic Backward Stochastic Differential Equations High-dimensional

2020-10-07abs ↗pdf ↗

Deep Actor-Critic Learning for Distributed Power Control in Wireless Mobile Networkseess.SP

Deep actor-critic learning optimizes power control in mobile networks.

problem Optimizing power control in large-scale wireless mobile networks.

method Multi-agent deep reinforcement learning with deep deterministic policy gradient.

result The algorithm maximizes a global utility function in a distributed manner.

power control wireless networks deep reinforcement learning actor-critic deep deterministic policy gradient

2020-09-14abs ↗pdf ↗

Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycs.LG

Single-timescale actor-critic finds globally optimal policy.

problem Finding globally optimal policy in reinforcement learning.

method Simultaneous actor and critic updates with linear or deep neural network approximations.

result Actor sequence converges to globally optimal policy at

O(K^{-1/2})

rate.

actor-critic reinforcement learning globally optimal policy single-timescale sublinear convergence

2020-08-02abs ↗pdf ↗

Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximationcs.LG

Study efficient convergence of RL algorithm with function approximation.

problem Convergence of actor-critic algorithm with nonlinear function approximation.

method Stochastic gradient descent ascent with adaptive proximal term, Polyak-Łojasiewicz condition.

result First efficient convergence result with rate of O(sqrt{ln(N d G^2) / N}).

Reinforcement Learning Actor-Critic Nonlinear Function Approximation Convergence Analysis Polyak-Łojasiewicz Condition

2022-02-28abs ↗pdf ↗

Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensuscs.LG

In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional…

2019-03-21abs ↗pdf ↗

Improving Actor-Critic Training with Steerable Action-Value Approximation Errorscs.LG

USAC balances pessimism and optimism in actor-critic training for better exploration and performance.

problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.

method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.

result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.

actor-critic reinforcement learning exploration optimism pessimism

2024-06-06abs ↗pdf ↗

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcs.LG

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…

2018-01-04abs ↗pdf ↗

A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learningcs.LG

This paper considers a distributed reinforcement learning problem in which a network of multiple agents aim to cooperatively maximize the globally averaged return through communication with only local neighbors. A randomized communication-efficient multi-agent actor-critic algorithm is proposed for possibly unidirectio…

2019-07-06abs ↗pdf ↗

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learningcs.LG

This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-…

2019-03-15abs ↗pdf ↗

Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithmscs.LG

This paper improves sample complexity for AC and NAC algorithms under Markovian sampling.

problem Improving sample complexity for actor-critic and natural actor-critic algorithms.

method Characterizes convergence rate and sample complexity under Markovian sampling and mini-batch data.

result Improves sample complexity for AC and NAC algorithms by orders of magnitude.

actor-critic natural actor-critic reinforcement learning sample complexity Markovian sampling

2020-04-27abs ↗pdf ↗

Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscs.LG

Study policy gradient and actor-critic methods for continuous-time reinforcement learning.

problem Continuous-time reinforcement learning with policy gradient and actor-critic approaches.

method Regularized exploratory formulation, martingale approach, simultaneous policy and value function updates.

result Proposed two types of actor-critic algorithms for online and offline learning.

reinforcement learning policy gradient actor-critic continuous time martingale

2021-11-22abs ↗pdf ↗

Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Spacecs.LG

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

2019-03-04abs ↗pdf ↗

Adversarially Guided Actor-Criticcs.LG

AGAC uses an adversary to enhance exploration in complex tasks.

problem Sample inefficiency in complex environments, especially in tasks requiring efficient exploration.

method Integrates an adversary into the actor-critic framework to encourage innovative exploration strategies.

result AGAC leads to more exhaustive exploration and outperforms state-of-the-art methods.

reinforcement learning actor-critic exploration adversarial sample inefficiency

2021-02-08abs ↗pdf ↗

Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcs.LG

Paper provides convergence guarantees for off-policy NAC with finite sample complexity.

problem Convergence analysis of off-policy natural actor-critic algorithm.

method Finite-sample analysis with Importance Sampling and Q-trace algorithm.

result Converges to global optimal policy with sample complexity

\mathcal{O}(ε^{-3}\log^2(1/ε))

natural actor-critic off-policy finite-sample convergence Importance Sampling

2021-02-18abs ↗pdf ↗

From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Predictioncs.CL

In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning. Inspired by the connection, we propose two sequence prediction algorithms, …

2018-04-29abs ↗pdf ↗

Distributional Advantage Actor-Criticcs.LG

In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optima…

2018-06-10abs ↗pdf ↗

Self-Imitation Learningcs.LG

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL sig…

2018-06-14abs ↗pdf ↗