arXiv research

New self-imitation learning method improves performance in continuous control tasks.

LBQL improves Q-learning by using lookahead bounds for better performance.

CQL learns conservative Q-functions to improve offline RL performance.

New method estimates optimal Q-values with better accuracy for specific problems.

The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.

GUM tackles MARL by avoiding overestimation through state-marginal restriction.

We introduce and analyze a form of variance-reduced $Q$-learning. For $γ$-discounted MDPs with finite state space $\mathcal{X}$ and action space $\mathcal{U}$, we prove that it yields an $ε$-accurate estimate of the optimal $Q$-function in the $\ell_\infty$-norm using $\mathcal{O} \left(\left(\frac{D}{ ε^2 (1-γ)^3} \ri…

A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. \cite{jin2018q} proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP. In this paper, we adapt Q-learning with…

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.

Study Q-learning with averaging for reinforcement learning, proving efficient inference and error bounds.

UCBMQ improves Q-learning by adding momentum to correct bias and limit regret.

Logarithmic regret achieved in Q-learning with positive gap.

Sample complexity bounds are a common performance metric in the Reinforcement Learning literature. In the discounted cost, infinite horizon setting, all of the known bounds have a factor that is a polynomial in $1/(1-γ)$, where $γ< 1$ is the discount factor. For a large discount factor, these bounds seem to imply that …

The current paper studies the problem of agnostic $Q$-learning with function approximation in deterministic systems where the optimal $Q$-function is approximable by a function in the class $\mathcal{F}$ with approximation error $δ\ge 0$. We propose a novel recursion-based algorithm and show that if $δ= O\left(ρ/\sqrt{…

We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the difficulty of running fully adaptive algorithms in real-world applications (such as medical domains), and …

Deep Q-Learning models optimal exercise strategies for option-type products.

In an episodic Markov Decision Process (MDP) problem, an online algorithm chooses from a set of actions in a sequence of $H$ trials, where $H$ is the episode length, in order to maximize the total payoff of the chosen actions. Q-learning, as the most popular model-free reinforcement learning (RL) algorithm, directly pa…

Double Q-learning has the same mean-squared error as Q-learning under certain conditions.

DE-QT detects optimal Q-learning stopping points.

VO$Q$L optimizes RL with sparse rewards using weighted bounds.

A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.

This paper analyzes momentum Q-learning with finite-sample guarantees.

EBQL reduces bias in Q-learning for improved performance.

Linear Q-learning converges to a bounded set without divergence.

Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…

Q-learning for average cost MDPs gets a concentration bound.

New algorithm uses Whittle index to improve Q-learning for restless bandits.

This paper analyzes how periodic and soft target updates stabilize linear Q-learning.

The use of target networks is a common practice in deep reinforcement learning for stabilizing the training; however, theoretical understanding of this technique is still limited. In this paper, we study the so-called periodic Q-learning algorithm (PQ-learning for short), which resembles the technique used in deep Q-le…

Consider a Markov decision process (MDP) that admits a set of state-action features, which can linearly express the process's probabilistic transition model. We propose a parametric Q-learning algorithm that finds an approximate-optimal policy using a sample size proportional to the feature dimension $K$ and invariant …

Kernelized Q-learning achieves good performance with minimal data.

Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.

New algorithm reduces sample and communication complexities in federated Q-learning.

This paper improves Q-learning bounds using reference-advantage decomposition.

Study Whittle index learning algorithms for restless bandits with constant stepsizes.

Extended analysis of Q-learning's efficiency, matching optimal regret.

Proposes a new Q-learning method for survival outcomes in clinical trials.

Paper analyzes finite-time convergence of double Q-learning.

A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.

Robust $Q$-learning for mean-field control under Wasserstein uncertainty

Q-learning is one of the most popular methods in Reinforcement Learning (RL). Transfer Learning aims to utilize the learned knowledge from source tasks to help new tasks to improve the sample complexity of the new tasks. Considering that data collection in RL is both more time and cost consuming and Q-learning converge…

This paper introduces sample-averaged Q-learning for better RL performance.

Improved sample complexity for target Q-learning in finite MDPs with generative oracle.

Proposes a robust Q-learning method to improve treatment strategy estimation.

Proposes a deep spectral Q-learning for mobile health data.

New Q-learning algorithms reduce regret in inventory control problems.

Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.

We consider model-free reinforcement learning for infinite-horizon discounted Markov Decision Processes (MDPs) with a continuous state space and unknown transition kernel, when only a single sample path under an arbitrary policy of the system is available. We consider the Nearest Neighbor Q-Learning (NNQL) algorithm to…