New self-imitation learning method improves performance in continuous control tasks.
problem Improving off-policy learning in continuous control tasks.
method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.
result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.
LBQL improves Q-learning by using lookahead bounds for better performance.
problem Improving Q-learning in stochastic environments.
method LBQL uses lookahead bounds to construct dual penalties and track upper and lower bounds via stochastic approximation.
result LBQL converges faster and is more robust to hyperparameters than standard Q-learning.
CQL learns conservative Q-functions to improve offline RL performance.
problem Leveraging large, static datasets in reinforcement learning without further interaction.
method Conservative Q-learning (CQL) which learns a conservative Q-function to lower-bound policy values.
result CQL substantially outperforms existing offline RL methods, often achieving 2-5 times higher final returns.
New method estimates optimal Q-values with better accuracy for specific problems.
problem Estimating optimal Q-values in reinforcement learning is difficult and varies by problem instance.
method Local minimax framework and variance-reduced Q-learning.
result Sharp lower bounds on estimation accuracy for Q-learning.
The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.
problem Analyzing the cumulative regret of Nash Q-learning in 2-player turn-based stochastic Markov games.
method Proposed gap-dependent logarithmic upper bounds for cumulative regret in episodic tabular setting and discounted game setting.
result The proposed bounds match theoretical lower bounds up to a logarithmic term.
GUM tackles MARL by avoiding overestimation through state-marginal restriction.
problem Overestimation of values in large joint state-action spaces.
method Greedy UnMixing through state-marginal restriction and unmixing.
result Superior performance compared to existing Q-learning and general MARL algorithms.
New Q-learning algorithm reduces sample complexity for large discount factors.
problem Large discount factors make Q-learning algorithms inefficient.
method Introduces a new Q-learning algorithm with uniformly bounded sample complexity.
result The new algorithm achieves asymptotic covariance that is a quadratic in 1 / ( 1 − ρ ∗ γ ) 1/(1- ρ^* γ) 1/ ( 1 − ρ ∗ γ ) . We introduce and analyze a form of variance-reduced Q Q Q -learning. For γ γ γ -discounted MDPs with finite state space X \mathcal{X} X and action space U \mathcal{U} U , we prove that it yields an ε ε ε -accurate estimate of the optimal Q Q Q -function in the ℓ ∞ \ell_\infty ℓ ∞ -norm using $\mathcal{O} \left(\left(\frac{D}{ ε^2 (1-γ)^3} \ri…
A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. \cite{jin2018q} proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP. In this paper, we adapt Q-learning with…
Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.
problem Understanding the sample complexity of Q-learning in synchronous settings.
method Analyzing Q-learning in synchronous MDPs with state and action spaces, proving minimax optimal sample complexity for TD learning and Q-learning under certain conditions.
result Q-learning requires more samples than minimax bounds suggest, revealing strict sub-optimality when action space is more than one.
Study Q-learning with averaging for reinforcement learning, proving efficient inference and error bounds.
problem Efficient inference and error bounds for Q-learning with averaging.
method Functional central limit theorem and asymptotic linear estimator for optimal Q-value function.
result Standardized partial-sum process converges weakly to a rescaled Brownian motion, matching instance-dependent lower bound for error.
UCBMQ improves Q-learning by adding momentum to correct bias and limit regret.
problem Improving Q-learning's bias and regret in reinforcement learning.
method UCBMQ combines Q-learning with an upper confidence bound and momentum term.
result UCBMQ guarantees a regret of O ( H 3 S A T + H 4 S A ) O(\sqrt{H^3SAT}+ H^4 S A ) O ( H 3 S A T + H 4 S A ) with a linear second-order term in S S S . Logarithmic regret achieved in Q-learning with positive gap.
problem Achieving logarithmic cumulative regret in Q-learning with positive sub-optimality gap.
method Optimistic Q-learning with logarithmic regret bound.
result Logarithmic cumulative regret bound proven for optimistic Q-learning.
New algorithm finds optimal policy with polynomial trajectories in deterministic systems.
problem Finding optimal policy in deterministic systems with function approximation.
method Novel recursion-based algorithm with tight bounds on error and sample complexity.
result Optimal policy found using O ( dim E ) O(\dim_E) O ( dim E ) trajectories with $δ= O\left(ρ/\sqrt{\dim_E}
ight)$ . We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the difficulty of running fully adaptive algorithms in real-world applications (such as medical domains), and …
Deep Q-Learning models optimal exercise strategies for option-type products.
problem Modeling optimal exercise strategies for option-type products.
method Reinforcement learning approach using deep neural networks to approximate the Q-function.
result Pricing the contract at inception and deriving bounds on the option price.
In an episodic Markov Decision Process (MDP) problem, an online algorithm chooses from a set of actions in a sequence of H H H trials, where H H H is the episode length, in order to maximize the total payoff of the chosen actions. Q-learning, as the most popular model-free reinforcement learning (RL) algorithm, directly pa…
Double Q-learning has the same mean-squared error as Q-learning under certain conditions.
problem Comparing the mean-squared error of Double Q-learning and Q-learning.
method Theoretical analysis based on Lyapunov equations for both tabular and linear function approximation settings.
result The asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning under specific conditions.
VO Q Q Q L optimizes RL with sparse rewards using weighted bounds.
problem Sparse rewards and non-linear function approximation in RL.
method VO Q Q Q L combines Q Q Q -learning with weighted bounds for optimal regret. result Achieves asymptotically optimal regret for linear function approximation.
DE-QT detects optimal Q-learning stopping points.
problem Information loss in Q-learning during prolonged training.
method Introducing DE-QT to detect entropy changes in Q-tables.
result DE-QT identifies the best stopping point for Q-learning.
A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.
problem Stabilizing model-free off-policy deep reinforcement learning with soft divergence.
method Representing past experiences as a QGraph, selecting a subgraph with favorable structure, and using lower bounds for temporal difference learning.
result QG-DDPG method is less prone to soft divergence and more robust to hyperparameters.
This paper analyzes momentum Q-learning with finite-sample guarantees.
problem Improving Q-learning performance with momentum schemes.
method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.
result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.
EBQL reduces bias in Q-learning for improved performance.
problem Over- and under-estimation biases in Q-learning degrade performance.
method Ensemble Bootstrapping to reduce both over- and under-estimation biases.
result EBQL outperforms other Q-learning methods in Atari games.
PQ-learning improves Q-learning by periodically updating target estimates.
problem Improving sample complexity in Q-learning for finding optimal policies.
method Maintains two Q-value estimates, one online and one target, updated periodically.
result PQ-learning achieves better sample complexity for finding epsilon-optimal policies.
Linear Q-learning converges to a bounded set without divergence.
problem Proving linear Q-learning does not diverge and converges to a bounded set.
method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.
result First L 2 L^2 L 2 convergence rate of linear Q-learning iterates to a bounded set. Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…
Q-learning for average cost MDPs gets a concentration bound.
problem Finding bounds for Q-learning in average cost MDPs.
method Derives a concentration bound using shortest path problem equivalence.
result Numerical comparison with relative value iteration shows the bound's effectiveness.
New algorithm uses Whittle index to improve Q-learning for restless bandits.
problem Optimizing decision-making in multiarmed restless bandits with average reward.
method Introduces a novel reinforcement learning algorithm combining Q-learning and Whittle index policy.
result Demonstrates significant computational gains and excellent empirical performance.
This paper analyzes how periodic and soft target updates stabilize linear Q-learning.
problem Theoretical explanation of stabilization mechanisms for linear Q-learning.
method Exact analysis using switched linear system dynamics and the joint spectral radius.
result Periodic and soft target updates can guarantee convergence to the exact projected Q-Bellman solution under specific conditions.
Consider a Markov decision process (MDP) that admits a set of state-action features, which can linearly express the process's probabilistic transition model. We propose a parametric Q-learning algorithm that finds an approximate-optimal policy using a sample size proportional to the feature dimension K K K and invariant …
Kernelized Q-learning achieves good performance with minimal data.
problem Efficient Q-learning in high-dimensional spaces.
method Kernelized Q-learning framework with effective dimensionality.
result Concrete regret bounds for linear and Gaussian RBF kernels.
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
New algorithm reduces sample and communication complexities in federated Q-learning.
problem Optimal Q-function learning in federated Q-learning with limited communication.
method Introduced Fed-DVR-Q algorithm for order-optimal sample and communication complexities.
result Complete characterization of sample-communication complexity trade-off.
This paper improves Q-learning bounds using reference-advantage decomposition.
problem Improving Q-learning bounds in MDPs with positive suboptimality gaps.
method Develops a novel error decomposition framework to prove gap-dependent regret bounds.
result Establishes logarithmic gap-dependent regret bounds for Q-learning.
Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
Extended analysis of Q-learning's efficiency, matching optimal regret.
problem Theoretical guarantees of Q-learning's efficiency and optimal regret.
method Survey of related research, detailed proof reasoning.
result Q-learning with UCB exploration achieves sample efficiency matching optimal regret.
Proposes a new Q-learning method for survival outcomes in clinical trials.
problem Incomplete follow-up data and nonlinear covariate effects in clinical trials.
method Combines Buckley-James boosting with flexible base learners for estimating optimal treatment regimes.
result Improves treatment decision accuracy and stability in longitudinal clinical trials.
AQL uses amortized inference to handle high-dimensional action spaces in Q-learning.
problem Difficulty in maximizing over large action spaces in Q-learning.
method Replace expensive maximization over all actions with a maximization over a small subset sampled from a learned proposal distribution.
result AQL outperforms existing methods on continuous control tasks with up to 21 dimensional actions.
Paper analyzes finite-time convergence of double Q-learning.
problem Overestimation issue in Q-learning.
method Finite-time analysis of double Q-learning.
result Convergence to ε-accurate neighborhood in finite iterations.
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
Robust Q Q Q -learning for mean-field control under Wasserstein uncertainty
problem Mean-field control under Wasserstein uncertainty
method Quantization-and-projection scheme with Wasserstein dual reformulation
result Convergence and finite-time iteration bounds
Q-learning is one of the most popular methods in Reinforcement Learning (RL). Transfer Learning aims to utilize the learned knowledge from source tasks to help new tasks to improve the sample complexity of the new tasks. Considering that data collection in RL is both more time and cost consuming and Q-learning converge…
This paper introduces sample-averaged Q-learning for better RL performance.
problem Improving reinforcement learning algorithms by managing uncertainty.
method Integrates statistical inference into Q-learning through sample averaging and functional central limit theorem.
result Establishes a unified theoretical foundation for sample-averaged Q-learning.
Improved sample complexity for target Q-learning in finite MDPs with generative oracle.
problem Sample complexity of target Q-learning in finite MDPs with a generative oracle.
method Analyzed target Q-learning algorithm in tabular case with a generative oracle, improved sample complexity.
result Improved sample complexity for target Q-learning in various scenarios.
Proposes a robust Q-learning method to improve treatment strategy estimation.
problem Misspecification of working models in Q-learning leads to confounding and efficiency loss.
method Uses data-adaptive techniques to estimate nuisance parameters robustly.
result Asymptotic behavior of robust Q-learning estimators is studied and shown to be useful.
A new Q-learning controller improves line follower robot control.
problem Challenges in controlling line follower robots due to unknown mechanical characteristics and uncertainties.
method Simulated annealing based Q learning method to address controller performance issues.
result The proposed controller outperforms conventional P controllers in line follower robots.
Proposes a deep spectral Q-learning for mobile health data.
problem Personalized treatment assignment for patients with time-varying covariates.
method Integrates PCA with deep Q-learning for mixed frequency data.
result Mean return converges to optimal under estimated optimal policy.
New Q-learning algorithms reduce regret in inventory control problems.
problem Efficiently learning optimal policies in inventory control problems with limited feedback.
method Proposed Elimination-Based Half-Q-Learning (HQL) and Full-Q-Learning (FQL) algorithms with theoretical regret bounds.
result HQL incurs i l d e O ( H 3 T ) ilde{\mathcal{O}}(H^3\sqrt{ T}) i l d e O ( H 3 T ) regret, FQL incurs i l d e O ( H 2 T ) ilde{\mathcal{O}}(H^2\sqrt{ T}) i l d e O ( H 2 T ) regret, independent of state and action space sizes.