On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “upper confidence bound momentum Q-learning”

UCB Momentum Q-learning: Correcting the bias without forgettingstat.ML

UCBMQ improves Q-learning by adding momentum to correct bias and limit regret.

problem Improving Q-learning's bias and regret in reinforcement learning.

method UCBMQ combines Q-learning with an upper confidence bound and momentum term.

result UCBMQ guarantees a regret of

O(\sqrt{H^3SAT}+ H^4 S A )

with a linear second-order term in

S

reinforcement learning Q-learning momentum upper confidence bound regret

2021-03-01abs ↗pdf ↗

When a Reinforcement Learning Agent Encounters Unknown Unknownscs.LG

AI agent learns to handle unknown unknown states in reinforcement learning.

problem Handling unexpected, previously unseen states in reinforcement learning.

method Proposes EMDP-GA model with NIVE approach to expand value functions.

result Asymptotically consistent regret and comparable computational complexity.

reinforcement learning unknown unknowns episodic Markov decision process upper confidence bound momentum Q-learning asymptotically consistent

2025-05-19abs ↗pdf ↗

Momentum Q-learning with Finite-Sample Convergence Guaranteecs.LG

This paper analyzes momentum Q-learning with finite-sample guarantees.

problem Improving Q-learning performance with momentum schemes.

method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.

result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.

Q-learning momentum finite-sample analysis function approximation convergence rate

2020-07-30abs ↗pdf ↗

UCB Exploration via Q-Ensemblescs.LG

We show how an ensemble of $Q^*$ -functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$ -learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experimen…

2017-06-05abs ↗pdf ↗

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximationcs.LG

New algorithm reduces RL complexity with low switching costs.

problem Exploration-exploitation dilemma in RL with complex models.

method Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation.

result Achieves minimax optimal regret of

O(d\sqrt{HK})

and near-optimal policy switching cost.

reinforcement learning upper confidence bound function approximation exploration-exploitation minimax optimal

2023-11-26abs ↗pdf ↗

Lookahead-Bounded Q-Learningcs.LG

LBQL improves Q-learning by using lookahead bounds for better performance.

problem Improving Q-learning in stochastic environments.

method LBQL uses lookahead bounds to construct dual penalties and track upper and lower bounds via stochastic approximation.

result LBQL converges faster and is more robust to hyperparameters than standard Q-learning.

Q-learning lookahead bounds stochastic environments convergence robustness

2020-06-28abs ↗pdf ↗

Statistically Model Checking PCTL Specifications on Markov Decision Processes via Reinforcement Learningcs.LG

Statistical model checking for PCTL on MDPs using reinforcement learning.

problem Model checking PCTL specifications on MDPs with statistical methods.

method Reinforcement learning for policy search, statistical model checking with UCB-based Q-learning.

result Provably guaranteed statistical model checking method for PCTL specifications on MDPs.

reinforcement learning statistical model checking Markov decision processes probabilistic CTL policy search

2020-04-01abs ↗pdf ↗

Optimizing Returns Using the Hurst Exponent and Q Learning on Momentum and Mean Reversion Strategiesq-fin.ST

Optimizes trading returns using Hurst exponent and Q-learning.

problem Maximizing returns from momentum and mean reversion strategies.

method Classifies assets using Hurst exponent and uses Q-learning to improve trading algorithms.

result Trading with Hurst exponent can achieve higher returns but at higher risk.

Hurst exponent Q-learning momentum mean reversion trading strategies

2022-05-23abs ↗pdf ↗

Single-partition adaptive Q-learningcs.LG

SPAQL improves RL by adaptively partitioning state-action space and learning a time-invariant policy.

problem Efficient model-free reinforcement learning with scalable algorithms.

method Adaptive Q-learning with UCB and Boltzmann exploration, automatically tuning temperature.

result SPAQL converges faster and uses fewer resources than AQL, showing higher sample efficiency.

reinforcement learning adaptive Q-learning UCB exploration Boltzmann exploration scalable algorithms

2020-07-14abs ↗pdf ↗

Exploration via Epistemic Value Estimationcs.LG

Proposes EVE for efficient exploration in reinforcement learning.

problem Efficient exploration in reinforcement learning.

method EVE: a recipe for posterior over parameters, facilitating efficient exploration.

result Competitive performance on benchmarks, efficient exploration confirmed.

reinforcement learning exploration epistemic value sequential decision making neural networks

2023-03-07abs ↗pdf ↗

The Efficacy of Pessimism in Asynchronous Q-Learningcs.LG

The paper improves Q-learning by incorporating pessimism for better sample efficiency.

problem Improving sample efficiency in asynchronous Q-learning with non-i.i.d. data.

method Developed an algorithmic framework that incorporates the principle of pessimism into asynchronous Q-learning, penalizing infrequently-visited state-action pairs based on suitable lower confidence bounds (LCBs).

result Achieved near-optimal sample complexity, providing theoretical support for the use of pessimism in non-i.i.d. data.

Q-learning Asynchronous Pessimism Lower Confidence Bounds Sample Efficiency

2022-03-14abs ↗pdf ↗

Gap-Dependent Bounds for Two-Player Markov Gamescs.LG

The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.

problem Analyzing the cumulative regret of Nash Q-learning in 2-player turn-based stochastic Markov games.

method Proposed gap-dependent logarithmic upper bounds for cumulative regret in episodic tabular setting and discounted game setting.

result The proposed bounds match theoretical lower bounds up to a logarithmic term.

Q-learning Markov Games Cumulative Regret Nash Q-learning Logarithmic Bounds

2021-07-01abs ↗pdf ↗

On Lai's Upper Confidence Bound in Multi-Armed Banditsstat.ML

The paper honors Lai's contributions to multi-armed bandits and establishes new regret bounds.

problem Improving regret bounds in multi-armed bandit problems.

method Establishes non-asymptotic regret bounds for upper confidence bound indices.

result New regret bounds match Lai-Robbins lower bound.

multi-armed bandits upper confidence bound regret bounds Tze Leung Lai

2024-10-03abs ↗pdf ↗

Asymptotic Analysis of Sample-averaged Q-learningcs.LG

This paper introduces sample-averaged Q-learning for better RL performance.

problem Improving reinforcement learning algorithms by managing uncertainty.

method Integrates statistical inference into Q-learning through sample averaging and functional central limit theorem.

result Establishes a unified theoretical foundation for sample-averaged Q-learning.

reinforcement learning Q-learning statistical inference functional central limit theorem batch scheduling

2024-10-14abs ↗pdf ↗

Online Statistical Inference of Constant Sample-averaged Q-Learningstat.ML

The paper proposes a method to infer Q-values online with Q-Learning.

problem High variance and instability in reinforcement learning algorithms.

method Adapting FCLT for a modified Q-learning approach and constructing confidence intervals.

result The proposed method provides more stable and reliable inference of Q-values.

reinforcement learning Q-learning statistical inference functional central limit theorem confidence intervals

2026-03-27abs ↗pdf ↗

Improved Regret Bounds for Gaussian Process Upper Confidence Bound in Bayesian Optimizationcs.LG

Paper improves regret bounds for Gaussian process upper confidence bound in Bayesian optimization.

problem Minimizing regret in Gaussian process bandit optimization.

method Gaussian process upper confidence bound (GP-UCB) algorithm with refined analysis.

result Achieves

O(\sqrt{T \ln^2 T})

cumulative regret under squared exponential kernel.

Bayesian optimization Gaussian process Upper confidence bound Regret bounds Matérn kernel

2025-06-02abs ↗pdf ↗

Differentiable Linear Bandit Algorithmcs.LG

A new differentiable UCB algorithm for linear bandits learns adaptive confidence bounds.

problem Inability of UCB to strike optimal exploration-exploitation due to confidence bounds.

method Proposes a differentiable linear bandit algorithm and a gradient estimator for learning adaptive confidence bounds.

result Achieves a

ilde{\mathcal{O}}(\hatβ\sqrt{dT})

upper bound of

T

-round regret.

bandit algorithms confidence bounds differentiability gradient ascent regret minimization

2020-06-04abs ↗pdf ↗

Information-Directed Exploration for Deep Reinforcement Learningcs.LG

Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropri…

2018-12-18abs ↗pdf ↗

Bayesian estimation of probabilities of default for low default portfoliosq-fin.RM

The estimation of probabilities of default (PDs) for low default portfolios by means of upper confidence bounds is a well established procedure in many financial institutions. However, there are often discussions within the institutions or between institutions and supervisors about which confidence level to use for the…

2011-12-23abs ↗pdf ↗

Regret Analysis for Randomized Gaussian Process Upper Confidence Boundcs.LG

This paper improves GP-UCB by using a shifted exponential distribution for confidence parameters.

problem Theoretical confidence parameter in GP-UCB increases with iterations, leading to large values.

method Introduced IRGP-UCB, a randomized variant of GP-UCB using a shifted exponential distribution for confidence parameters.

result IRGP-UCB achieves sub-linear regret without increasing the confidence parameter.

Bayesian optimization Gaussian process Upper confidence bound Regret analysis Randomization

2024-09-02abs ↗pdf ↗

Agnostic Q-learning with Function Approximation in Deterministic Systems: Tight Bounds on Approximation Error and Sample Complexitycs.LG

New algorithm finds optimal policy with polynomial trajectories in deterministic systems.

problem Finding optimal policy in deterministic systems with function approximation.

method Novel recursion-based algorithm with tight bounds on error and sample complexity.

result Optimal policy found using

O(\dim_E)

trajectories with $δ= O\left(ρ/\sqrt{\dim_E} ight)$.

Q-learning function approximation deterministic systems sample complexity Eluder dimension

2020-02-17abs ↗pdf ↗

On the Generalization of Stochastic Gradient Descent with Momentumcs.LG

New insights into training machine learning models with momentum.

problem Lack of theoretical understanding on the generalization error of momentum-based methods.

method Analyzed modified momentum-based update rule (SGDEM) for smooth Lipschitz loss functions.

result SGDEM admits an upper-bound on the generalization error for smooth Lipschitz loss functions.

Stochastic Gradient Descent Momentum Generalization Error Machine Learning Convex Loss Functions

2021-02-26abs ↗pdf ↗

Transition Transfer

Q

-Learning for Composite Markov Decision Processesstat.ML

UCB-TQL learns from multiple tasks with shared dynamics and adapts to task-specific variations.

problem Transfer reinforcement learning with composite MDPs where tasks share core dynamics but have sparse differences.

method UCB-TQL, a novel transfer RL algorithm for composite MDPs.

result Achieved a regret bound of

ilde{O}(\sqrt{eH^5N})

that scales independently of the ambient dimension.

transfer reinforcement learning composite MDPs UCB-TQL regret bound shared dynamics

2025-02-01abs ↗pdf ↗

Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datamath.ST

Study on statistical inference for nonlinear stochastic approximation with Markovian data.

problem Statistical inference for nonlinear stochastic approximation algorithms with Markovian data.

method Established a functional central limit theorem for the partial-sum process of the target parameter estimate, providing asymptotic pivotal statistics for constructing confidence intervals.

result Valid and efficient asymptotic inference method for nonlinear stochastic approximation algorithms with Markovian data.

statistical inference nonlinear stochastic approximation Markovian data functional central limit theorem asymptotic pivotal statistic

2023-02-15abs ↗pdf ↗

Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Controlcs.LG

New algorithm tackles non-stationary RL with near-optimal regret bounds.

problem Model-free reinforcement learning in non-stationary Markov decision processes.

method Proposed RestartQ-UCB algorithm with Freedman-type bonus terms.

result Achieves near-optimal dynamic regret bound in non-stationary RL.

Reinforcement Learning Non-stationary Model-free Upper Confidence Bounds Multi-Agent RL

2020-10-07abs ↗pdf ↗

SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcs.LG

SUNRISE improves off-policy RL algorithms by integrating ensemble methods.

problem Stability and exploration issues in off-policy RL algorithms.

method SUNRISE combines ensemble-based weighted Bellman backups and upper-confidence bounds for efficient exploration.

result SUNRISE improves the performance of off-policy RL algorithms across various domains.

reinforcement learning ensemble methods off-policy deep learning exploration

2020-07-09abs ↗pdf ↗

Q-learning with Uniformly Bounded Variance: Large Discounting is Not a Barrier to Fast Learningcs.LG

New Q-learning algorithm reduces sample complexity for large discount factors.

problem Large discount factors make Q-learning algorithms inefficient.

method Introduces a new Q-learning algorithm with uniformly bounded sample complexity.

result The new algorithm achieves asymptotic covariance that is a quadratic in

1/(1- ρ^* γ)

Q-learning Uniformly bounded variance Large discounting Reinforcement Learning Sample complexity

2020-02-24abs ↗pdf ↗

A Note on the Equivalence of Upper Confidence Bounds and Gittins Indices for Patient Agentscs.LG

This note gives a short, self-contained, proof of a sharp connection between Gittins indices and Bayesian upper confidence bound algorithms. I consider a Gaussian multi-armed bandit problem with discount factor $γ$ . The Gittins index of an arm is shown to equal the $γ$ -quantile of the posterior distribution of the arm'…

2019-04-09abs ↗pdf ↗

Randomised Gaussian Process Upper Confidence Bound for Bayesian Optimisationcs.LG

Improved Bayesian optimisation method using randomised Gaussian process UCB.

problem Improving performance in Bayesian optimisation.

method Developed a modified Gaussian process upper confidence bound (GP-UCB) acquisition function.

result The method achieves better performance than GP-UCB in various problems.

Bayesian optimisation Gaussian process Upper Confidence Bound Randomised sampling Exploration-exploitation trade-off

2020-06-08abs ↗pdf ↗

Accelerating Federated Learning via Momentum Gradient Descentcs.LG

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update w…

2019-10-08abs ↗pdf ↗

Accelerated Target Updates for Q-learningcs.LG

This paper studies accelerations in Q-learning algorithms. We propose an accelerated target update scheme by incorporating the historical iterates of Q functions. The idea is conceptually inspired by the momentum-based accelerated methods in the optimization theory. Conditions under which the proposed accelerated algor…

2019-05-07abs ↗pdf ↗

High-dimensional near-optimal experiment design for drug discovery via Bayesian sparse samplingcs.LG

Bayesian methods improve drug discovery experiment design.

problem Optimizing drug screening experiments in high-dimensional data.

method Bayesian inference and optimisation with upper confidence bound algorithms, Thompson sampling, and sparse tree search.

result Sparse tree search techniques outperform other methods in drug toxicity screening.

drug discovery experiment design Bayesian inference Thompson sampling upper confidence bound

2021-04-23abs ↗pdf ↗

One-bit feedback is sufficient for upper confidence bound policiescs.LG

One-bit feedback suffices for a bandit problem's optimal strategy.

problem Optimal strategy for multi-armed bandit problem with limited feedback.

method Coding and decoding schemes for one-bit feedback to mimic full-reward feedback.

result Regret ratio approaches 1 with one-bit feedback.

multi-armed bandit one-bit feedback upper confidence bound regret

2020-12-04abs ↗pdf ↗

Recommendation System-based Upper Confidence Bound for Online Advertisingcs.IR

In this paper, the method UCB-RS, which resorts to recommendation system (RS) for enhancing the upper-confidence bound algorithm UCB, is presented. The proposed method is used for dealing with non-stationary and large-state spaces multi-armed bandit problems. The proposed method has been targeted to the problem of the …

2019-09-09abs ↗pdf ↗

On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimizationstat.ML

Momentum SGD fails to track nonstationary optima due to drift amplification.

problem Tracking nonstationary optima in stochastic optimization.

method Theoretical analysis of SGD and momentum variants under strong convexity and smoothness.

result Momentum incurs a drift-amplification penalty that diverges as the momentum parameter approaches 1, leading to systematic lag.

momentum SGD nonstationary optimization drift amplification stochastic optimization gradient descent

2026-01-18abs ↗pdf ↗

Bayesian Optimization of Robustness Measures under Input Uncertainty: A Randomized Gaussian Process Upper Confidence Bound Approachstat.ML

A new method optimizes robustness measures under input uncertainty using randomized Gaussian process upper confidence bound.

problem Optimizing robustness measures under input uncertainty.

method Randomized robustness measure GP-UCB (RRGP-UCB) that samples β from a chi-squared-based distribution.

result RRGP-UCB provides tight bounds on expected regret.

Bayesian optimization robustness measures input uncertainty Gaussian process upper confidence bound

2025-04-04abs ↗pdf ↗

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycs.LG

Pessimistic Q-learning improves sample efficiency in offline reinforcement learning.

problem Insufficient coverage and sample scarcity in offline reinforcement learning datasets.

method Pessimistic Q-learning algorithm for offline reinforcement learning, focusing on variance reduction.

result Near-optimal sample complexity achieved with the proposed algorithm.

offline reinforcement learning pessimism Q-learning sample efficiency Markov decision processes

2022-02-28abs ↗pdf ↗

On Regret Bounds of Thompson Sampling for Bayesian Optimizationstat.ML

This paper analyzes regret bounds for Gaussian process Thompson sampling.

problem Analyzing the performance of Gaussian process Thompson sampling (GP-TS) in Bayesian optimization.

method The paper derives several regret bounds for GP-TS, including a lower bound, upper bounds on the second moment of cumulative regret, expected lenient regret, and improved cumulative regret.

result The paper provides improved regret upper bounds for GP-TS, showing that it suffers from a polynomial dependence on

1/δ

with probability

δ

Bayesian optimization Thompson sampling Gaussian process regret bounds upper confidence bound

2026-03-10abs ↗pdf ↗

Finite-Time Logarithmic Bayes Regret Upper Boundscs.LG

New bounds for Bayesian bandits show prior improves performance.

problem Improving regret bounds for Bayesian bandits.

method Upper confidence bound algorithm with finite-time logarithmic regret bounds.

result Derives

O(c_Δ\log n)

and

O(c_h \log^2 n)

upper bounds for Bayesian bandits.

Bayesian bandits logarithmic regret upper confidence bound prior information multi-armed bandits

2023-06-15abs ↗pdf ↗

Bootstrapping Upper Confidence Boundstat.ML

Upper Confidence Bound (UCB) method is arguably the most celebrated one used in online decision making with partial information feedback. Existing techniques for constructing confidence bounds are typically built upon various concentration inequalities, which thus lead to over-exploration. In this paper, we propose a n…

2019-06-12abs ↗pdf ↗

Optimistic Q-learning for average reward and episodic reinforcement learningcs.LG

New algorithm for average reward learning with bounded hitting time assumption.

problem Minimizing regret in average reward reinforcement learning with bounded hitting time.

method Optimistic Q-learning with a novel

\overline{L}

operator for bounded hitting time.

result Regret bound of

ilde{O}(H^5 S\sqrt{AT})

for average reward learning.

reinforcement learning average reward bounded hitting time Q-learning episodic

2024-07-18abs ↗pdf ↗

A Finite Sample Complexity Bound for Distributionally Robust Q-learningcs.LG

Improved reinforcement learning for environments with distributional shifts.

problem Learning optimal policies in environments with distributional shifts.

method Distributionally robust Q-learning with multi-level Monte Carlo estimator.

result Proved upper bound on sample complexity for robust RL.

reinforcement learning distributionally robust Q-learning sample complexity Monte Carlo

2023-02-26abs ↗pdf ↗

Lower Bounds for the Area of Black Holes in Terms of Mass, Charge, and Angular Momentumgr-qc

The most general formulation of Penrose's inequality yields a lower bound for ADM mass in terms of the area, charge, and angular momentum of black holes. This inequality is in turn equivalent to an upper and lower bound for the area in terms of the remaining quantities. In this note, we establish the lower bound for a …

2013-06-20abs ↗pdf ↗

Q

L: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcs.LG

VO $Q$ L optimizes RL with sparse rewards using weighted bounds.

problem Sparse rewards and non-linear function approximation in RL.

method VO

Q

L combines

Q

-learning with weighted bounds for optimal regret.

result Achieves asymptotically optimal regret for linear function approximation.

Reinforcement Learning Optimal Regret Sparse Rewards Non-linear Approximation Weighted Bounds

2022-12-12abs ↗pdf ↗

Bayesian Bandit Algorithms with Approximate Inference in Stochastic Linear Banditsstat.ML

Bayesian bandit algorithms with approximate inference improve regret bounds in stochastic linear bandits.

problem Theoretical justification for Bayesian bandit algorithms with approximate inference in stochastic linear bandits.

method Proposed a theoretical framework to analyze approximate inference impact and conducted frequentist regret analysis on LinTS and LinBUCB.

result LinTS and LinBUCB preserve their original regret upper bounds with larger constant terms in approximate inference settings.

Bayesian bandit stochastic linear bandits approximate inference regret analysis Thompson Sampling

2024-06-20abs ↗pdf ↗

Self-Imitation Learning via Generalized Lower Bound Q-learningcs.LG

New self-imitation learning method improves performance in continuous control tasks.

problem Improving off-policy learning in continuous control tasks.

method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.

result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.

self-imitation learning lower-bound Q-learning continuous control n-step Q-learning performance improvement

2020-06-12abs ↗pdf ↗

Concentration bounds for SSP Q-learning for average cost MDPscs.LG

Q-learning for average cost MDPs gets a concentration bound.

problem Finding bounds for Q-learning in average cost MDPs.

method Derives a concentration bound using shortest path problem equivalence.

result Numerical comparison with relative value iteration shows the bound's effectiveness.

Q-learning Markov decision processes concentration bounds shortest path problem

2022-06-07abs ↗pdf ↗

On the Generalization of Stochastic Gradient Descent with Momentumcs.LG

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that there exists a convex loss function for which the stability gap fo…

2018-09-12abs ↗pdf ↗