Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
Optimistic Q-learning reduces sample complexity for systems with known dynamics.
problem Sample efficiency in reinforcement learning with partial dynamics knowledge.
method Optimistic Q-learning algorithm for systems with additive disturbance model.
result Achieves i l d e O ( e x t P o l y ( H ) T ) ilde{\mathcal{O}}( ext{Poly}(H)\sqrt{T}) i l d e O ( e x t P o l y ( H ) T ) regret under perfect knowledge of dynamics. New algorithm for average reward learning with bounded hitting time assumption.
problem Minimizing regret in average reward reinforcement learning with bounded hitting time.
method Optimistic Q-learning with a novel L ‾ \overline{L} L operator for bounded hitting time. result Regret bound of i l d e O ( H 5 S A T ) ilde{O}(H^5 S\sqrt{AT}) i l d e O ( H 5 S A T ) for average reward learning. Logarithmic regret achieved in Q-learning with positive gap.
problem Achieving logarithmic cumulative regret in Q-learning with positive sub-optimality gap.
method Optimistic Q-learning with logarithmic regret bound.
result Logarithmic cumulative regret bound proven for optimistic Q-learning.
RandQL is a new model-free algorithm for MDPs with a novel learning rate randomization approach.
problem Minimizing regret in episodic MDPs with model-free methods.
method Randomized Q-learning with learning rate randomization.
result RandQL achieves optimal regret bounds in both tabular and metric state-action spaces.
Extended Q-learning stability and convergence analysis.
problem Stability and convergence of Q-learning with linear function approximation.
method Analysis of ( ε , κ ) (\varepsilon,κ) ( ε , κ ) -tamed Gibbs policy, and other Q-learning variants. result Far more structure is required for convergence of Q-learning.
VO Q Q Q L optimizes RL with sparse rewards using weighted bounds.
problem Sparse rewards and non-linear function approximation in RL.
method VO Q Q Q L combines Q Q Q -learning with weighted bounds for optimal regret. result Achieves asymptotically optimal regret for linear function approximation.
We analyze reinforcement learning algorithms using a distributional approach.
problem Theoretical analysis of reinforcement learning algorithms for constant step-sizes.
method Distributional approach to theoretical analyses of reinforcement learning algorithms.
result TD( λ λ λ ) and Q Q Q -Learning have contractive update rules in the space of distributions of functions, leading to exponentially fast convergence. Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-p…
We establish that an optimistic variant of Q-learning applied to a fixed-horizon episodic Markov decision process with an aggregated state representation incurs regret O ~ ( H 5 M K + ε H K ) \tilde{\mathcal{O}}(\sqrt{H^5 M K} + εHK) O ~ ( H 5 M K + ε H K ) , where H H H is the horizon, M M M is the number of aggregate states, K K K is the number of episodes, and ε ε ε is …
Improved Q-learning for multi-agent reinforcement learning by weighting joint action values.
problem QMIX restricts Q Q Q -values to monotonic mixtures, limiting complex value functions. method Introduced weighted projection to recover optimal policies, improving performance.
result CW QMIX and OW QMIX outperform baseline QMIX on multi-agent tasks.
Paper optimizes reinforcement learning in self-play games with reduced steps.
problem Optimizing reinforcement learning algorithms for self-play in two-player zero-sum games.
method Proposes optimistic Nash Q-learning and Nash V-learning algorithms with improved sample complexity.
result Achieves sample complexity of O ( S A B ) O(SAB) O ( S A B ) for Nash Q-learning and O ( S ( A + B ) ) O(S(A+B)) O ( S ( A + B )) for Nash V-learning, closing the gap with lower bounds. Double Q-learning has the same mean-squared error as Q-learning under certain conditions.
problem Comparing the mean-squared error of Double Q-learning and Q-learning.
method Theoretical analysis based on Lyapunov equations for both tabular and linear function approximation settings.
result The asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning under specific conditions.
Bayesian classifier improves robustness with optimistic score ratio.
problem Limited information on class-conditional distribution.
method Optimistic score ratio for robust binary classification.
result Bayesian classifier using optimistic score ratio is robust and computationally tractable.
We show that the optimistic limits of the colored Jones polynomials of the hyperbolic knots coincide with the optimistic limits of the Kashaev invariants modulo 4 π 2 4π^2 4 π 2 .
We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning. Optimistic approaches presented in the literature apply an optimistic boost to the value estimate at each state-action pair and select actions that are greedy with respect to the resulting optimistic value f…
DE-QT detects optimal Q-learning stopping points.
problem Information loss in Q-learning during prolonged training.
method Introducing DE-QT to detect entropy changes in Q-tables.
result DE-QT identifies the best stopping point for Q-learning.
The optimistic limit is the mathematical formulation of the classical limit which is a physical method to expect the actual limit by using saddle point method of certain potential function. The original optimistic limit of the Kashaev invariant was formulated by Yokota, and a modified formulation was suggested by the a…
This paper analyzes momentum Q-learning with finite-sample guarantees.
problem Improving Q-learning performance with momentum schemes.
method Proposes MomentumQ algorithm integrating Nesterov and Polyak's momentum schemes, analyzes convergence for function approximations.
result Establishes finite-sample convergence rates for MomentumQ, demonstrating better performance than vanilla Q-learning.
EBQL reduces bias in Q-learning for improved performance.
problem Over- and under-estimation biases in Q-learning degrade performance.
method Ensemble Bootstrapping to reduce both over- and under-estimation biases.
result EBQL outperforms other Q-learning methods in Atari games.
PQ-learning improves Q-learning by periodically updating target estimates.
problem Improving sample complexity in Q-learning for finding optimal policies.
method Maintains two Q-value estimates, one online and one target, updated periodically.
result PQ-learning achieves better sample complexity for finding epsilon-optimal policies.
Optimistic estimate predicts best fitting performance of nonlinear models.
problem Evaluating the potential of nonlinear models in fitting.
method Proposes an optimistic estimate to quantify the smallest sample size for fitting nonlinear models.
result Predicts specific subsets of targets that can be fitted at overparameterization.
New self-imitation learning method improves performance in continuous control tasks.
problem Improving off-policy learning in continuous control tasks.
method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.
result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.
Linear Q-learning converges to a bounded set without divergence.
problem Proving linear Q-learning does not diverge and converges to a bounded set.
method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.
result First L 2 L^2 L 2 convergence rate of linear Q-learning iterates to a bounded set. Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…
New algorithm reduces regret in sequential decision-making problems.
problem Balancing exploration and exploitation in online sequential decision problems.
method Variational Bayesian optimistic sampling (VBOS) for optimizing policies.
result VBOS achieves i l d e O ( A T ) ilde O(\sqrt{AT}) i l d e O ( A T ) Bayesian regret for stochastic multi-armed bandits. Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.
problem Understanding the sample complexity of Q-learning in synchronous settings.
method Analyzing Q-learning in synchronous MDPs with state and action spaces, proving minimax optimal sample complexity for TD learning and Q-learning under certain conditions.
result Q-learning requires more samples than minimax bounds suggest, revealing strict sub-optimality when action space is more than one.
Q-learning for average cost MDPs gets a concentration bound.
problem Finding bounds for Q-learning in average cost MDPs.
method Derives a concentration bound using shortest path problem equivalence.
result Numerical comparison with relative value iteration shows the bound's effectiveness.
Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
New algorithm uses Whittle index to improve Q-learning for restless bandits.
problem Optimizing decision-making in multiarmed restless bandits with average reward.
method Introduces a novel reinforcement learning algorithm combining Q-learning and Whittle index policy.
result Demonstrates significant computational gains and excellent empirical performance.
Characterizes concept classes for optimistic online learning.
problem Understanding minimal assumptions for online learnability.
method Investigates two questions about concept classes' learnability.
result Characterizes all concept classes for optimistically universal online learnability.
LBQL improves Q-learning by using lookahead bounds for better performance.
problem Improving Q-learning in stochastic environments.
method LBQL uses lookahead bounds to construct dual penalties and track upper and lower bounds via stochastic approximation.
result LBQL converges faster and is more robust to hyperparameters than standard Q-learning.
This paper analyzes how periodic and soft target updates stabilize linear Q-learning.
problem Theoretical explanation of stabilization mechanisms for linear Q-learning.
method Exact analysis using switched linear system dynamics and the joint spectral radius.
result Periodic and soft target updates can guarantee convergence to the exact projected Q-Bellman solution under specific conditions.
We show how to take any two parameter-free online learning algorithms with different regret guarantees and obtain a single algorithm whose regret is the minimum of the two base algorithms. Our method is embarrassingly simple: just add the iterates. This trick can generate efficient algorithms that adapt to many norms s…
Kernelized Q-learning achieves good performance with minimal data.
problem Efficient Q-learning in high-dimensional spaces.
method Kernelized Q-learning framework with effective dimensionality.
result Concrete regret bounds for linear and Gaussian RBF kernels.
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
New algorithm reduces sample and communication complexities in federated Q-learning.
problem Optimal Q-function learning in federated Q-learning with limited communication.
method Introduced Fed-DVR-Q algorithm for order-optimal sample and communication complexities.
result Complete characterization of sample-communication complexity trade-off.
New algorithms reduce forecasting errors by leveraging optimistic learning and hinting.
problem Real-time climate forecasting with delayed feedback.
method Developed optimistic online learning algorithms that reduce regret under delayed feedback.
result Optimistic learning algorithms achieve optimal regret guarantees under delayed feedback.
This paper improves Q-learning bounds using reference-advantage decomposition.
problem Improving Q-learning bounds in MDPs with positive suboptimality gaps.
method Develops a novel error decomposition framework to prove gap-dependent regret bounds.
result Establishes logarithmic gap-dependent regret bounds for Q-learning.
Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
Extended analysis of Q-learning's efficiency, matching optimal regret.
problem Theoretical guarantees of Q-learning's efficiency and optimal regret.
method Survey of related research, detailed proof reasoning.
result Q-learning with UCB exploration achieves sample efficiency matching optimal regret.
Proposes a new Q-learning method for survival outcomes in clinical trials.
problem Incomplete follow-up data and nonlinear covariate effects in clinical trials.
method Combines Buckley-James boosting with flexible base learners for estimating optimal treatment regimes.
result Improves treatment decision accuracy and stability in longitudinal clinical trials.
Omega method mitigates noise in stochastic game optimization.
problem Noise sensitivity and convergence issues in stochastic game optimization.
method Omega method incorporates EMA of historic gradients in its update rule.
result Omega method outperforms optimistic gradient method in stochastic games.
New method minimizes regret in AMDP with high probability.
problem Pessimistic regret analysis in AMDP.
method Developed a new policy search method achieving optimistic regret.
result Achieved sublinear optimistic regret with high probability.
Paper analyzes finite-time convergence of double Q-learning.
problem Overestimation issue in Q-learning.
method Finite-time analysis of double Q-learning.
result Convergence to ε-accurate neighborhood in finite iterations.
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
Robust Q Q Q -learning for mean-field control under Wasserstein uncertainty
problem Mean-field control under Wasserstein uncertainty
method Quantization-and-projection scheme with Wasserstein dual reformulation
result Convergence and finite-time iteration bounds
Q-learning is one of the most popular methods in Reinforcement Learning (RL). Transfer Learning aims to utilize the learned knowledge from source tasks to help new tasks to improve the sample complexity of the new tasks. Considering that data collection in RL is both more time and cost consuming and Q-learning converge…