On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,694 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Asynchronous Q-learning”

Central Limit Theorems for Asynchronous Averaged Q-Learningcs.LG

Paper proves CLTs for Q-learning with asynchronous updates.

problem Establishing convergence rates for Q-learning algorithms.

method Polyak-Ruppert averaging, non-asymptotic and functional CLTs.

result Convergence rates in Wasserstein distance for Q-learning.

Q-learning asynchronous updates central limit theorems Polyak-Ruppert averaging Wasserstein distance

2025-09-23abs ↗pdf ↗

Robust

Q

-learning for mean-field control under Wasserstein uncertainty in common noisemath.OC

Robust $Q$ -learning for mean-field control under Wasserstein uncertainty

problem Mean-field control under Wasserstein uncertainty

method Quantization-and-projection scheme with Wasserstein dual reformulation

result Convergence and finite-time iteration bounds

$Q$-learning mean-field control Wasserstein uncertainty common noise systemic risk

2026-06-18abs ↗pdf ↗

The Efficacy of Pessimism in Asynchronous Q-Learningcs.LG

The paper improves Q-learning by incorporating pessimism for better sample efficiency.

problem Improving sample efficiency in asynchronous Q-learning with non-i.i.d. data.

method Developed an algorithmic framework that incorporates the principle of pessimism into asynchronous Q-learning, penalizing infrequently-visited state-action pairs based on suitable lower confidence bounds (LCBs).

result Achieved near-optimal sample complexity, providing theoretical support for the use of pessimism in non-i.i.d. data.

Q-learning Asynchronous Pessimism Lower Confidence Bounds Sample Efficiency

2022-03-14abs ↗pdf ↗

Gaussian Approximation for Asynchronous Q-learningstat.ML

The paper analyzes Q-learning convergence rates with asynchronous updates.

problem Analyzing convergence rates of asynchronous Q-learning algorithms.

method Derives rates of convergence using high-dimensional central limit theorems.

result Establishes a rate of order up to

n^{-1/6} \log^{4} (nS A)

for hyper-rectangles.

Q-learning asynchronous updates convergence rates central limit theorem Markov chains

2026-04-08abs ↗pdf ↗

Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncs.LG

Asynchronous Q-learning achieves optimal Q-function estimation with reduced sample complexity.

problem Learning the optimal Q-function in asynchronous Q-learning with limited samples.

method Demonstrates sample complexity bound with variance reduction for

γ

-discounted MDPs.

result Sample complexity improved by a factor of

|\mathcal{S}||\mathcal{A}|

and

t_{mix}|\mathcal{S}||\mathcal{A}|

Asynchronous Q-learning Sample Complexity Markov Decision Process Variance Reduction Finite-Time Analysis

2020-06-04abs ↗pdf ↗

The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcs.LG

Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.

problem Collaborative learning in distributed RL settings with limited data sharing.

method Analyzes synchronous and asynchronous federated Q-learning, proposes importance averaging.

result Achieves linear speedup with heterogeneity, robust to local trajectory heterogeneity.

federated learning reinforcement learning Q-learning heterogeneity asynchronous

2023-05-18abs ↗pdf ↗

Achieving

\varepsilon^{-2}

Dependence for Average-Reward Q-Learning with a New Contraction Principlecs.LG

New Q-learning method achieves optimal sample complexity for average-reward problems.

problem Challenges in achieving optimal sample complexity for average-reward Q-learning.

method Synchronous and asynchronous Q-learning with a new contraction principle.

result Optimal

\widetilde{O}(\varepsilon^{-2})

sample complexity guarantees.

Q-learning Markov decision processes Contraction principle Average-reward Sample complexity

2026-01-29abs ↗pdf ↗

A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscs.LG

Unified framework for finite-sample RL algorithms using Lyapunov theory.

problem Finite-sample convergence guarantees of asynchronous RL algorithms.

method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.

result Mean-square error bounds and convergence for various RL algorithms.

Reinforcement Learning Asynchronous Algorithms Lyapunov Theory Finite-Sample Guarantees Mean-Square Error

2021-02-02abs ↗pdf ↗

Whittle Index Learning Algorithms for Restless Bandits with Constant Stepsizescs.LG

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.

method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.

result The algorithms learn the Whittle index effectively.

Whittle index restless bandits Q-learning constant stepsizes DQN

2024-09-06abs ↗pdf ↗

From Set Convergence to Pointwise Convergence: Finite-Time Guarantees for Average-Reward Q-Learning with Adaptive Stepsizescs.LG

This work analyzes $Q$ -learning with adaptive stepsizes for finite-time convergence.

problem Finite-time convergence analysis for average-reward

Q

-learning with adaptive stepsizes.

method Adaptive stepsizes as local clocks, time-inhomogeneous Markovian reformulation, almost-sure time-varying bounds, conditioning arguments, and Markov chain concentration inequalities.

result Convergence rates of

ilde{\mathcal{O}}(1/k)

for mean-square and pointwise mean-square convergence.

Q-learning adaptive stepsizes finite-time analysis average-reward convergence rates

2025-04-25abs ↗pdf ↗

Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Marketsq-fin.TR

RL agent learns to place limit orders for trading signals in financial markets.

problem Training an RL agent to execute trading signals in limit order book markets.

method Deep Duelling Double Q-learning with APEX architecture, using synthetic alpha signals.

result RL agent outperforms heuristic trading strategies in inventory management and order placing.

deep reinforcement learning limit order book markets trading signals Q-learning reinforcement learning

2023-01-20abs ↗pdf ↗

Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolationstat.ML

Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.

problem Understanding and optimizing Q-learning with constant stepsize.

method Connecting Q-learning to a Markov chain, proving distributional convergence and bias, applying Richardson-Romberg extrapolation.

result Explicit expression for the linear coefficient of the asymptotic bias and improvement of RR extrapolation method.

Q-learning constant stepsize distributional convergence bias extrapolation

2024-01-25abs ↗pdf ↗

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SAstat.ML

Study on nonsmooth contractive SA with constant stepsize and Q-learning.

problem Understanding convergence and bias in nonsmooth contractive SA with different noise types.

method Proposed prelimit coupling technique for steady-state convergence and derived asymptotic bias.

result Asymptotic bias of nonsmooth SA is proportional to the square root of the stepsize.

nonsmooth contractive SA Q-learning steady-state convergence asymptotic bias stepsize

2024-04-09abs ↗pdf ↗

Combining policy gradient and Q-learningcs.LG

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…

2016-11-05abs ↗pdf ↗

Finite-Time Analysis for Double Q-learningcs.LG

Paper analyzes finite-time convergence of double Q-learning.

problem Overestimation issue in Q-learning.

method Finite-time analysis of double Q-learning.

result Convergence to ε-accurate neighborhood in finite iterations.

finite-time analysis double Q-learning reinforcement learning stochastic approximation overestimation

2020-09-29abs ↗pdf ↗

Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysisstat.ML

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.

problem Understanding the sample complexity of Q-learning in synchronous settings.

method Analyzing Q-learning in synchronous MDPs with state and action spaces, proving minimax optimal sample complexity for TD learning and Q-learning under certain conditions.

result Q-learning requires more samples than minimax bounds suggest, revealing strict sub-optimality when action space is more than one.

Q-learning Markov decision process sample complexity minimax bounds TD learning

2021-02-12abs ↗pdf ↗

The Role of Target Update Frequencies in Q-Learningcs.LG

This work analyzes how often to update the target network in Q-learning.

problem Understanding the optimal frequency of target network updates in Q-learning.

method Formulated target updates as a nested optimization scheme, derived finite-time convergence analysis.

result Optimal target update frequency increases geometrically over time.

Q-learning target network target update frequency approximate dynamic programming finite-time convergence

2026-02-03abs ↗pdf ↗

On Gaussian approximation for entropy-regularized Q-learning with function approximationstat.ML

The paper analyzes the convergence rates of Q-learning with entropy regularization and linear function approximation.

problem Analyzing the convergence rates of Q-learning with entropy regularization and linear function approximation.

method The paper derives rates of convergence using the high-dimensional central limit theorem, linearization of the soft Bellman recursion, and Gaussian approximation for the leading martingale term.

result The algorithm's last iterate satisfies high-order moment bounds, with a Gaussian approximation bound of order

n^{-1/4}

Q-learning entropy regularization linear function approximation convergence rates Gaussian approximation

2026-05-17abs ↗pdf ↗

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision processeess.SY

Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.

problem Optimality in safety-constrained Markov decision processes with multichain structure.

method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.

result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.

Markov decision process Reinforcement learning Safety constraints Optimality Multi-objective optimization

2023-02-25abs ↗pdf ↗

DASA: Delay-Adaptive Multi-Agent Stochastic Approximationcs.AI

DASA speeds up SA with delayed agents, achieving N-fold speedup.

problem Speeding up Stochastic Approximation with asynchronous delays.

method DASA: Delay-Adaptive Multi-Agent Stochastic Approximation algorithm.

result First algorithm with convergence rate dependent on mixing time and average delay.

Stochastic Approximation Multi-Agent Systems Asynchronous Delays Convergence Rate Markov Chains

2024-03-25abs ↗pdf ↗

Finite-Time Bounds for Two-Time-Scale Stochastic Approximation with Arbitrary Norm Contractions and Markovian Noisecs.LG

New bounds for SA with arbitrary norm contractions and Markovian noise.

problem Finite-time analysis of two-time-scale stochastic approximation with arbitrary norm contractions and Markovian noise.

method Use of generalized Moreau envelope for arbitrary norm contractions and solutions of Poisson equation for Markovian noise.

result Mean square error decays at rates of

O(1/n^{2/3})

and

O(1/n)

under different conditions.

Stochastic Approximation Two-Time-Scale Arbitrary Norm Contractions Markovian Noise Finite-Time Bounds

2025-03-24abs ↗pdf ↗

The Mean-Squared Error of Double Q-Learningcs.LG

Double Q-learning has the same mean-squared error as Q-learning under certain conditions.

problem Comparing the mean-squared error of Double Q-learning and Q-learning.

method Theoretical analysis based on Lyapunov equations for both tabular and linear function approximation settings.

result The asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning under specific conditions.

mean-squared error Double Q-learning Q-learning Lyapunov equations asymptotic analysis

2020-07-09abs ↗pdf ↗

Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexitycs.LG

Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.

problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.

method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.

result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.

Asynchronous SGD Optimal Time Complexity Heterogeneous Workers Machine Learning Distributed Computing

2025-01-27abs ↗pdf ↗

Q-Learning with Differential Entropy of Q-Tablescs.LG

DE-QT detects optimal Q-learning stopping points.

problem Information loss in Q-learning during prolonged training.

method Introducing DE-QT to detect entropy changes in Q-tables.

result DE-QT identifies the best stopping point for Q-learning.

Q-learning entropy information loss policy search differential entropy

2020-06-26abs ↗pdf ↗

Asynchronous ε-Greedy Bayesian Optimisationcs.LG

AEGiS optimizes expensive function evaluations asynchronously.

problem Optimizing expensive black-box functions efficiently.

method Asynchronous ε-Greedy Bayesian Optimisation combining greedy search and Thompson sampling.

result AEGiS outperforms existing asynchronous BO methods.

Bayesian optimisation asynchronous ε-Greedy Thompson sampling expensive function evaluation

2020-10-15abs ↗pdf ↗

Secure Aggregation for Buffered Asynchronous Federated Learningcs.LG

Secure aggregation for buffered asynchronous federated learning without TEEs.

problem Privacy and convergence in buffered asynchronous federated learning.

method Developed a new protocol (BASecAgg) that ensures privacy without TEEs by carefully designing masks.

result BASecAgg achieves similar convergence guarantees as FedBuff without TEEs.

Federated Learning Asynchronous Training Secure Aggregation Buffered Privacy

2021-10-05abs ↗pdf ↗

Momentum Q-learning with Finite-Sample Convergence Guaranteecs.LG

This paper analyzes momentum Q-learning with finite-sample guarantees.

problem Improving Q-learning performance with momentum schemes.

method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.

result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.

Q-learning momentum finite-sample analysis function approximation convergence rate

2020-07-30abs ↗pdf ↗

Ensemble Bootstrapping for Q-Learningcs.LG

EBQL reduces bias in Q-learning for improved performance.

problem Over- and under-estimation biases in Q-learning degrade performance.

method Ensemble Bootstrapping to reduce both over- and under-estimation biases.

result EBQL outperforms other Q-learning methods in Atari games.

Q-learning reinforcement learning bias reduction deep Q-learning ATARI games

2021-02-28abs ↗pdf ↗

Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimizationstat.ML

Standard acquisition functions are sufficient for asynchronous Bayesian optimization.

problem Redundant and repeated queries in asynchronous Bayesian optimization.

method Conceptual analysis and theoretical guarantees of standard acquisitions.

result Standard acquisition functions achieve theoretical guarantees equivalent to Thompson sampling in asynchronous settings.

Bayesian optimization asynchronous Upper Confidence Bound Thompson sampling diversity

2026-03-13abs ↗pdf ↗

Self-Imitation Learning via Generalized Lower Bound Q-learningcs.LG

New self-imitation learning method improves performance in continuous control tasks.

problem Improving off-policy learning in continuous control tasks.

method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.

result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.

self-imitation learning lower-bound Q-learning continuous control n-step Q-learning performance improvement

2020-06-12abs ↗pdf ↗

Linear

Q

-Learning Does Not Diverge in

L^2

: Convergence Rates to a Bounded Setcs.LG

Linear Q-learning converges to a bounded set without divergence.

problem Proving linear Q-learning does not diverge and converges to a bounded set.

method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.

result First

L^2

convergence rate of linear Q-learning iterates to a bounded set.

Q-learning linear function approximation convergence rate bounded set ε-softmax policy

2025-01-31abs ↗pdf ↗

A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcs.LG

Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…

2019-12-10abs ↗pdf ↗

Asynchronous Batch Bayesian Optimisation with Improved Local Penalisationstat.ML

Batch Bayesian optimisation (BO) has been successfully applied to hyperparameter tuning using parallel computing, but it is wasteful of resources: workers that complete jobs ahead of others are left idle. We address this problem by developing an approach, Penalising Locally for Asynchronous Bayesian Optimisation on $k$ …

2019-01-29abs ↗pdf ↗

Model-based Asynchronous Hyperparameter and Neural Architecture Searchcs.LG

Asynchronous method for hyperparameter and neural architecture search.

problem Efficiently searching for optimal hyperparameters and neural architectures.

method Model-based, asynchronous multi-fidelity method combining Hyperband and Gaussian process-based Bayesian optimization.

result Substantial speed-ups over current state-of-the-art methods on various benchmarks.

hyperparameter optimization neural architecture search asynchronous search Bayesian optimization multi-fidelity

2020-03-24abs ↗pdf ↗

Asynchronous and Stochastic Distributed Resource Allocationmath.OC

This work tackles resource allocation in asynchronous and stochastic systems.

problem Distributed resource allocation in asynchronous and stochastic settings.

method Approximate stochastic primal-dual approach with asynchronous updates.

result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of

O(1/t)

resource allocation asynchronous systems stochastic optimization distributed computing convergence analysis

2025-09-01abs ↗pdf ↗

Advances in Asynchronous Parallel and Distributed Optimizationcs.LG

Advances in asynchronous optimization methods for machine learning.

problem Efficiently solving large-scale optimization problems in machine learning.

method Asynchronous parallel and distributed optimization methods, accounting for information delays.

result Degree of asynchrony impacts convergence rates in stochastic optimization methods.

machine learning optimization asynchronous distributed stochastic

2020-06-24abs ↗pdf ↗

Concentration bounds for SSP Q-learning for average cost MDPscs.LG

Q-learning for average cost MDPs gets a concentration bound.

problem Finding bounds for Q-learning in average cost MDPs.

method Derives a concentration bound using shortest path problem equivalence.

result Numerical comparison with relative value iteration shows the bound's effectiveness.

Q-learning Markov decision processes concentration bounds shortest path problem

2022-06-07abs ↗pdf ↗

Improved asynchronous parallel optimization analysis for stochastic incremental methodsmath.OC

As datasets continue to increase in size and multi-core computer architectures are developed, asynchronous parallel optimization algorithms become more and more essential to the field of Machine Learning. Unfortunately, conducting the theoretical analysis asynchronous methods is difficult, notably due to the introducti…

2018-01-11abs ↗pdf ↗

Whittle index based Q-learning for restless bandits with average rewardcs.LG

New algorithm uses Whittle index to improve Q-learning for restless bandits.

problem Optimizing decision-making in multiarmed restless bandits with average reward.

method Introduces a novel reinforcement learning algorithm combining Q-learning and Whittle index policy.

result Demonstrates significant computational gains and excellent empirical performance.

Whittle index restless bandits Q-learning reinforcement learning average reward

2020-04-29abs ↗pdf ↗

Unbounded Gradients in Federated Learning with Buffered Asynchronous Aggregationcs.LG

FedBuff improves federated learning scalability with asynchronous updates.

problem Limited scalability of federated learning with synchronous updates.

method Introduces asynchronous updates (staleness) in federated learning.

result Theoretical analysis shows improved convergence rate with boundedness removed.

federated learning asynchronous updates scalability convergence rate secure aggregation

2022-10-03abs ↗pdf ↗

Asynchronous Stochastic Approximation with Differential Inclusionsstat.ML

The asymptotic pseudo-trajectory approach to stochastic approximation of Benaim, Hofbauer and Sorin is extended for asynchronous stochastic approximations with a set-valued mean field. The asynchronicity of the process is incorporated into the mean field to produce convergence results which remain similar to those of a…

2011-12-10abs ↗pdf ↗

Asynchronous stochastic convex optimizationmath.OC

We show that asymptotically, completely asynchronous stochastic gradient procedures achieve optimal (even to constant factors) convergence rates for the solution of convex optimization problems under nearly the same conditions required for asymptotic optimality of standard stochastic gradient procedures. Roughly, the n…

2015-08-04abs ↗pdf ↗

Gradient Energy Matching for Distributed Asynchronous Gradient Descentcs.LG

Distributed asynchronous SGD has become widely used for deep learning in large-scale systems, but remains notorious for its instability when increasing the number of workers. In this work, we study the dynamics of distributed asynchronous SGD under the lens of Lagrangian mechanics. Using this description, we introduce …

2018-05-22abs ↗pdf ↗

Lookahead-Bounded Q-Learningcs.LG

LBQL improves Q-learning by using lookahead bounds for better performance.

problem Improving Q-learning in stochastic environments.

method LBQL uses lookahead bounds to construct dual penalties and track upper and lower bounds via stochastic approximation.

result LBQL converges faster and is more robust to hyperparameters than standard Q-learning.

Q-learning lookahead bounds stochastic environments convergence robustness

2020-06-28abs ↗pdf ↗

Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamicsstat.ML

This paper analyzes how periodic and soft target updates stabilize linear Q-learning.

problem Theoretical explanation of stabilization mechanisms for linear Q-learning.

method Exact analysis using switched linear system dynamics and the joint spectral radius.

result Periodic and soft target updates can guarantee convergence to the exact projected Q-Bellman solution under specific conditions.

Q-learning linear function approximation stabilization periodic updates soft updates

2026-05-31abs ↗pdf ↗

pySOT and POAP: An event-driven asynchronous framework for surrogate optimizationmath.OC

This paper describes Plumbing for Optimization with Asynchronous Parallelism (POAP) and the Python Surrogate Optimization Toolbox (pySOT). POAP is an event-driven framework for building and combining asynchronous optimization strategies, designed for global optimization of expensive functions where concurrent function …

2019-07-30abs ↗pdf ↗

Second-Order Convergence of Asynchronous Parallel Stochastic Gradient Descent: When Is the Linear Speedup Achieved?cs.LG

In machine learning, asynchronous parallel stochastic gradient descent (APSGD) is broadly used to speed up the training process through multi-workers. Meanwhile, the time delay of stale gradients in asynchronous algorithms is generally proportional to the total number of workers, which brings additional deviation from …

2019-10-14abs ↗pdf ↗

Periodic Q-Learningcs.LG

The use of target networks is a common practice in deep reinforcement learning for stabilizing the training; however, theoretical understanding of this technique is still limited. In this paper, we study the so-called periodic Q-learning algorithm (PQ-learning for short), which resembles the technique used in deep Q-le…

2020-02-23abs ↗pdf ↗