On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Actor-Critic”

Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcs.LG

The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.

problem Lack of theoretical understanding of target networks in actor-critic methods.

method Proposes a theoretical analysis of an online target-based actor-critic algorithm with linear function approximation.

result Establishes asymptotic convergence results and finite-time analysis for both critic and actor.

Actor-critic Target networks Deep reinforcement learning Temporal difference learning Markovian sampling

2021-06-14abs ↗pdf ↗

A Finite Time Analysis of Two Time-Scale Actor Critic Methodscs.LG

This work analyzes actor-critic methods for faster convergence.

problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.

method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.

result Actor-critic method finds a first-order stationary point with

\mathcal{ ilde{O}}(ε^{-2.5})

sample complexity.

actor-critic two-time-scale finite-time analysis sample complexity reinforcement learning

2020-05-04abs ↗pdf ↗

On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Costcs.LG

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

2019-07-14abs ↗pdf ↗

PAC-Bayesian Soft Actor-Critic Learningcs.LG

New PAC-Bayesian approach stabilizes actor-critic learning.

problem Training instability in actor-critic algorithms.

method Employing PAC-Bayesian bound as the critic training objective.

result Significant improvement in online learning performance.

actor-critic reinforcement learning PAC-Bayesian Soft Actor-Critic stability

2023-01-30abs ↗pdf ↗

Finite-time analysis of single-timescale actor-criticcs.LG

New analysis shows actor-critic method converges efficiently in practical settings.

problem Understanding finite-time convergence of single-timescale actor-critic methods.

method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.

result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).

actor-critic finite-time analysis single-timescale reinforcement learning Markovian sampling

2022-10-18abs ↗pdf ↗

Variance Adjusted Actor Critic Algorithmsstat.ML

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…

2013-10-14abs ↗pdf ↗

Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learningcs.LG

WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.

problem Inherent instability in actor-critic reinforcement learning algorithms.

method Wasserstein adaptive value estimation with Sinkhorn approximation.

result Achieves $\mathcal{O}\left(\frac{1}{k} ight)$ convergence rate for critic's mean squared error.

reinforcement learning actor-critic Wasserstein stability Sinkhorn

2025-01-17abs ↗pdf ↗

Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycs.LG

Single-timescale actor-critic finds globally optimal policy.

problem Finding globally optimal policy in reinforcement learning.

method Simultaneous actor and critic updates with linear or deep neural network approximations.

result Actor sequence converges to globally optimal policy at

O(K^{-1/2})

rate.

actor-critic reinforcement learning globally optimal policy single-timescale sublinear convergence

2020-08-02abs ↗pdf ↗

Regret Analysis of Average-Reward Unichain MDPs via an Actor-Critic Approachcs.LG

Actor-Critic method achieves optimal regret for unichain MDPs.

problem Scalable regret analysis for infinite-horizon average-reward MDPs.

method NAC-B, a Natural Actor-Critic with batching.

result Order-optimal regret of

ilde{O}(\sqrt{T})

in infinite-horizon average-reward MDPs.

Actor-Critic Natural Actor-Critic MDP Regret Analysis Unichain

2025-05-26abs ↗pdf ↗

Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Methodcs.LG

A new method improves actor-critic RL by integrating HMC, enhancing policy distribution and exploration.

problem Actor-critic RL yields suboptimal policies due to amortization gap and insufficient exploration.

method Integrating Hamiltonian Monte Carlo (HMC) into the actor-critic RL framework.

result Improves policy distribution and exploration, leading to better policy estimates and higher returns.

actor-critic RL Hamiltonian Monte Carlo policy optimization reinforcement learning exploration

2021-03-22abs ↗pdf ↗

TT-DAC-PS: Twin-Target Deterministic Actor-Critic with Policy Smoothing for Optimal Trade Executioncs.AI

TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution

problem Optimal execution of large stock sell programs

method Twin-Target Deterministic Actor-Critic with Policy Smoothing

result Reduces mean implementation shortfall percentage

trade execution stock sell programs reinforcement learning optimization

2026-06-07abs ↗pdf ↗

Variance Reduction in Actor Critic Methods (ACM)cs.LG

After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the $L^2$ norm for the control variate estimators spanned by functions conditioned by the current state and acti…

2019-07-23abs ↗pdf ↗

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processescs.LG

Optimistic actor-critic tackles linear MDPs with parametric policies.

problem Theoretical limitations of existing actor-critic methods for linear MDPs.

method Proposes an optimistic actor-critic framework with parametric log-linear policies and approximate Thompson sampling.

result Achieves state-of-the-art sample complexity in both on-policy and off-policy settings.

actor-critic linear MDPs parametric policies logit-matching Thompson sampling

2026-03-30abs ↗pdf ↗

Guide Actor-Critic for Continuous Controlstat.ML

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this pa…

2017-05-22abs ↗pdf ↗

A Theoretical Justification for Asymmetric Actor-Critic Algorithmscs.LG

Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

problem Lack of precise theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.

method Adapting a finite-time convergence analysis to the asymmetric actor-critic setting with linear function approximators.

result A finite-time bound reveals that the asymmetric critic eliminates aliasing errors in the agent state.

reinforcement learning asymmetric learning finite-time convergence actor-critic algorithms linear function approximators

2025-01-31abs ↗pdf ↗

Global Convergence of the ODE Limit for Online Actor-Critic Algorithms in Reinforcement Learningcs.LG

Actor-critic algorithms converge to an ODE as data samples change dynamically.

problem Challenging to mathematically analyze due to non-i.i.d. data samples.

method Proved convergence to an ODE using time rescaling and geometric ergodicity.

result Convergence to the ODE limit and its properties proven.

Actor-Critic Reinforcement Learning Ordinary Differential Equation Geometric Ergodicity Convergence Rate

2021-08-19abs ↗pdf ↗

TD-Regularized Actor-Critic Methodscs.LG

Actor-critic methods can achieve incredible performance on difficult reinforcement learning problems, but they are also prone to instability. This is partly due to the interaction between the actor and critic during learning, e.g., an inaccurate step taken by one of them might adversely affect the other and destabilize…

2018-12-19abs ↗pdf ↗

Soft Actor-Critic for Discrete Action Settingscs.LG

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to dis…

2019-10-16abs ↗pdf ↗

An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processescs.LG

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

problem Risk-sensitive cost criterion in Markov decision processes.

method Actor-critic algorithm with function approximation.

result Asymptotic convergence of the actor-critic algorithm.

actor-critic Markov decision processes risk-sensitive function approximation asymptotic convergence

2025-02-17abs ↗pdf ↗

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learningcs.LG

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …

2019-12-11abs ↗pdf ↗

Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscs.AI

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

2018-01-31abs ↗pdf ↗

Mean Actor Criticstat.ML

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. …

2017-09-01abs ↗pdf ↗

Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Spacecs.LG

In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…

2019-03-04abs ↗pdf ↗

Actor-Critic learning for mean-field control in continuous timestat.ML

Study uses actor-critic method for continuous-time mean-field control with entropy regularisation.

problem Continuous-time mean-field control in reinforcement learning.

method Actor-critic approach with entropy regularisation, value function alternation, and Wasserstein space parametrisation.

result Derives exact parametrisation of actor and critic functions in linear-quadratic mean-field framework.

actor-critic mean-field control reinforcement learning entropy regularisation Wasserstein space

2023-03-13abs ↗pdf ↗

A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Rewardstat.ML

We develop an off-policy actor-critic algorithm for learning an optimal policy from a training set composed of data from multiple individuals. This algorithm is developed with a view towards its use in mobile health.

2016-07-18abs ↗pdf ↗

Finite-Time Analysis of Natural Actor-Critic for POMDPscs.LG

Study on natural actor-critic for POMDPs with finite memory.

problem Learning in partially observed Markov decision processes with noisy observations.

method Finite actor-critic method with multi-step temporal difference learning.

result First non-asymptotic global convergence for POMDPs with function approximation.

actor-critic POMDPs finite memory temporal difference convergence

2022-02-20abs ↗pdf ↗

Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learningcs.LG

We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and R…

2019-02-14abs ↗pdf ↗

Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimizationcs.LG

Study visualizes actor-critic loss landscapes for inventory optimization.

problem Difficulties in solving multi-store dynamic inventory control problems.

method Low-dimensional visualizations of actor loss function.

result Loss landscapes favor optimal policies in reinforcement learning.

actor-critic methods reinforcement learning inventory optimization loss landscapes policy gradients

2020-09-04abs ↗pdf ↗

Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscs.LG

Paper introduces a meta-critic for accelerating off-policy actor-critic learning.

problem Improving sample efficiency in continuous control tasks.

method Meta-critic that meta-learns an additional loss for the actor.

result Online meta-critic learning leads to improved performance in various continuous control environments.

meta-critic actor-critic reinforcement learning sample efficiency continuous control

2020-03-11abs ↗pdf ↗

Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentumcs.LG

Improved sample complexity for actor-critic algorithms in MDPs.

problem Achieving optimal policies with limited data in reinforcement learning.

method Single-timescale actor-critic with STORM (STOchastic Recursive Momentum) and a sample buffer.

result Optimal sample complexity of

O(ε^{-2})

for

ε

-optimal policies.

actor-critic reinforcement learning sample complexity Markov decision processes Momentum

2026-02-02abs ↗pdf ↗

Improving Actor-Critic Training with Steerable Action-Value Approximation Errorscs.LG

USAC balances pessimism and optimism in actor-critic training for better exploration and performance.

problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.

method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.

result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.

actor-critic reinforcement learning exploration optimism pessimism

2024-06-06abs ↗pdf ↗

Actor-Critic Algorithm for High-dimensional Partial Differential Equationscs.LG

Deep learning model solves high-dimensional PDEs using Actor-Critic approach.

problem Solving high-dimensional nonlinear PDEs efficiently.

method Reformulated PDE into BSDE system, inspired by Actor-Critic algorithm for deep RL.

result Improved model with fewer parameters, faster convergence, and less hyperparameter tuning.

Deep Learning Partial Differential Equations Actor-Critic Backward Stochastic Differential Equations High-dimensional

2020-10-07abs ↗pdf ↗

FORK: A Forward-Looking Actor For Model-Free Reinforcement Learningcs.LG

FORK improves model-free reinforcement learning performance.

problem Improving model-free reinforcement learning performance.

method Introducing a new forward-looking Actor (FORK) for Actor-Critic algorithms.

result FORK significantly improves performance in various environments.

reinforcement learning actor-critic model-free performance improvement forward-looking

2020-10-04abs ↗pdf ↗

Off-Policy Actor-Critic with Shared Experience Replaycs.LG

We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ …

2019-09-25abs ↗pdf ↗

Efficient

Q

-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learningcs.LG

New algorithms improve robust reinforcement learning under uncertainty.

problem Robust reinforcement learning in MDPs with contamination.

method Non-asymptotic convergence analysis of

Q

-learning and actor-critic methods.

result Efficient algorithms learn robust policies with minimal samples.

reinforcement learning robustness Q-learning actor-critic Markov decision processes

2025-06-08abs ↗pdf ↗

Actor critic learning algorithms for mean-field control with moment neural networksstat.ML

New algorithm solves mean-field control problems using actor-critic learning with moment neural networks.

problem Solving mean-field control problems in continuous time reinforcement learning.

method Gradient-based policy and value function learning with moment neural networks on the Wasserstein space.

result Effective solution for diverse mean-field control problems, including multi-dimensional and nonlinear settings.

mean-field control actor-critic moment neural networks Wasserstein space reinforcement learning

2023-09-08abs ↗pdf ↗

Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methodscs.LG

A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.

problem Underestimation bias in deep actor-critic methods for reinforcement learning.

method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.

result Improves state-of-the-art performance on OpenAI Gym tasks.

Reinforcement Learning Deep Q-learning Actor-Critic Underestimation Bias Deep Reinforcement Learning

2021-09-22abs ↗pdf ↗

Weak Convergence Analysis of Online Neural Actor-Critic Algorithmscs.LG

Neural networks trained with actor-critic algorithms converge to ODEs under weak convergence analysis.

problem Challenges in convergence analysis due to changing data distributions in online learning.

method Geometric ergodicity of data samples, Poisson equation, weak convergence techniques.

result Actor and critic networks converge to solutions of ODEs with random initial conditions.

Neural networks Actor-critic Convergence analysis Weak convergence Geometric ergodicity

2024-03-25abs ↗pdf ↗

Optimistic critics can empower small actorscs.LG

Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.

problem Performance degradation and overfitting in actor-critic models with smaller actors.

method Broad empirical investigations and analyses of asymmetric actor-critic setups, exploring techniques to mitigate value underestimation.

result Value underestimation is a key cause of performance degradation in smaller actor-critic models, and the critic plays a crucial role in mitigating this.

actor-critic reinforcement learning asymmetric models performance degradation value underestimation

2025-06-01abs ↗pdf ↗

Actor-Critics Can Achieve Optimal Sample Efficiencystat.ML

New actor-critic algorithm achieves optimal sample efficiency in RL.

problem Achieving

ε

-optimal policies with minimal samples in RL.

method Integrates optimism, off-policy critic estimation, and rare-switching policy resets.

result Sample complexity of

O(dH^5 \log|\mathcal{A}|/ε^2 + d H^4 \log|\mathcal{F}|/ ε^2)

trajectories.

Actor-Critic Reinforcement Learning Optimism Off-Policy Rare-Switching

2025-05-06abs ↗pdf ↗

AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learningcs.LG

AACC improves RL performance in changing environments.

problem Deterioration of RL performance in real-world tasks.

method Formalizes CMDPs, proposes AACC for contextual RL.

result AACC outperforms baselines in various simulated environments.

Reinforcement Learning Contextual RL CMDPs Actor-Critic Generalization

2022-08-03abs ↗pdf ↗

DAC: The Double Actor-Critic Architecture for Learning Optionscs.LG

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yieldi…

2019-04-29abs ↗pdf ↗

PEARL: Preconditioner Enhancement through Actor-critic Reinforcement Learningcs.LG

PEARL uses reinforcement learning to improve matrix preconditioners.

problem Learning effective preconditioners for iterative solvers is challenging.

method PEARL employs an actor-critic reinforcement learning framework to learn preconditioners dynamically.

result PEARL outperforms traditional and neural preconditioners in flexibility and solving speed.

preconditioning reinforcement learning matrix solvers actor-critic preconditioner learning

2025-01-18abs ↗pdf ↗

On the Convergence of Single-Timescale Actor-Criticcs.LG

Improved convergence for actor-critic algorithms in MDPs.

problem Global convergence analysis for actor-critic algorithms in MDPs.

method Introduced an analytical framework to handle complex recursions, established convergence to ε-close globally optimal policy with improved sample complexity.

result Converges to ε-close globally optimal policy with sample complexity of O(ε^(-3)) compared to O(ε^(-2)) for ε-close stationary policy.

actor-critic Markov Decision Processes convergence global optimality sample complexity

2024-10-11abs ↗pdf ↗

Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensuscs.LG

In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional…

2019-03-21abs ↗pdf ↗

Deep Actor-Critic Learning for Distributed Power Control in Wireless Mobile Networkseess.SP

Deep actor-critic learning optimizes power control in mobile networks.

problem Optimizing power control in large-scale wireless mobile networks.

method Multi-agent deep reinforcement learning with deep deterministic policy gradient.

result The algorithm maximizes a global utility function in a distributed manner.

power control wireless networks deep reinforcement learning actor-critic deep deterministic policy gradient

2020-09-14abs ↗pdf ↗

Adversarially Guided Actor-Criticcs.LG

AGAC uses an adversary to enhance exploration in complex tasks.

problem Sample inefficiency in complex environments, especially in tasks requiring efficient exploration.

method Integrates an adversary into the actor-critic framework to encourage innovative exploration strategies.

result AGAC leads to more exhaustive exploration and outperforms state-of-the-art methods.

reinforcement learning actor-critic exploration adversarial sample inefficiency

2021-02-08abs ↗pdf ↗

Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Accesscs.LG

Enhances reinforcement learning with partial state information.

problem Improving learning under partial observability with limited privileged signals.

method Introduced informed asymmetric actor-critic framework that uses arbitrary state-dependent privileged signals.

result Unbiased policy gradient estimates with arbitrary privileged signals.

reinforcement learning asymmetric actor-critic privileged signals partial observability policy gradient

2025-09-30abs ↗pdf ↗