arXiv research

Tabular Q-learning outperforms advanced RL methods in monetary policy.

This paper analyzes momentum Q-learning with finite-sample guarantees.

The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.

Double Q-learning has the same mean-squared error as Q-learning under certain conditions.

Can simple algorithms with a good representation solve challenging reinforcement learning problems? In this work, we answer this question in the affirmative, where we take "simple learning algorithm" to be tabular Q-Learning, the "good representations" to be a learned state abstraction, and "challenging problems" to be…

The use of target networks is a common practice in deep reinforcement learning for stabilizing the training; however, theoretical understanding of this technique is still limited. In this paper, we study the so-called periodic Q-learning algorithm (PQ-learning for short), which resembles the technique used in deep Q-le…

Paper proposes methods to handle missing data in online RL, improving efficiency and uncertainty capture.

Zap Q-learning is a recent class of reinforcement learning algorithms, motivated primarily as a means to accelerate convergence. Stability theory has been absent outside of two restrictive classes: the tabular setting, and optimal stopping. This paper introduces a new framework for analysis of a more general class of r…

Linear Q-learning converges to a bounded set without divergence.

RandQL is a new model-free algorithm for MDPs with a novel learning rate randomization approach.

Model-free Reinforcement Learning (RL) algorithms such as Q-learning [Watkins, Dayan 92] have been widely used in practice and can achieve human level performance in applications such as video games [Mnih et al. 15]. Recently, equipped with the idea of optimism in the face of uncertainty, Q-learning algorithms [Jin, Al…

Distributional reinforcement learning (distributional RL) has seen empirical success in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation. However, there are many different ways in which one can leverage the distributional approach to reinforcement learning. In this paper, we p…

Improved sample complexity for target Q-learning in finite MDPs with generative oracle.

Delusional bias is a fundamental source of error in approximate Q-learning. To date, the only techniques that explicitly address delusion require comprehensive search using tabular value estimates. In this paper, we develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "…

Q($Δ$)-Learning improves Q-Learning by separating action-value functions into different time scales.

Study Q-learning with averaging for reinforcement learning, proving efficient inference and error bounds.

A new algorithm for deep Q-learning with robustness to state transition uncertainty.

Logarithmic regret achieved in Q-learning with positive gap.

This paper improves Q-learning bounds using reference-advantage decomposition.

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…

UCBMQ improves Q-learning by adding momentum to correct bias and limit regret.

Paper proposes FedQ-Advantage for federated Q-learning with near-optimal regret and low communication cost.

Introduces LTQL for factored policies in cooperative MARL.

This work analyzes how often to update the target network in Q-learning.

New method reduces regret and communication costs in federated Q-learning.

Extended Q-learning stability and convergence analysis.

Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.

Proposes Constrained Q-learning for reinforcement learning with constraints.

Full-sampling (e.g., Q-learning) and pure-expectation (e.g., Expected Sarsa) algorithms are efficient and frequently used techniques in reinforcement learning. Q$(σ,λ)$ is the first approach unifies them with eligibility trace through the sampling degree $σ$. However, it is limited to the tabular case, for large-scale …

Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.

A new RL method improves revenue management with delayed feedback.

Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algorithms do not use optimistic initialisation despite taking inspiration from these provably efficient …

A MARL system improves productivity on a metallurgical pickling line.

Federated Q-Learning achieves linear regret speedup with low communication cost.

Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized experience replay (PER) which samples important transitions more frequently. In this paper, we propo…

FedLCB-Q learns optimal policies from federated offline data with linear speedup.

All-goals updating exploits the off-policy nature of Q-learning to update all possible goals an agent could have from each transition in the world, and was introduced into Reinforcement Learning (RL) by Kaelbling (1993). In prior work this was mostly explored in small-state RL problems that allowed tabular representati…

Improved Q-learning for multi-agent reinforcement learning by weighting joint action values.

DE-QT detects optimal Q-learning stopping points.

EBQL reduces bias in Q-learning for improved performance.

Bayesian ATM improves stability and efficiency in mobile health interventions.

This paper improves reinforcement learning efficiency for large-scale MDPs.

New self-imitation learning method improves performance in continuous control tasks.

Fine-grained gap-dependent regret bounds for reinforcement learning.

Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.

Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it ta…

Q-learning for average cost MDPs gets a concentration bound.