arXiv research

New algorithm BEAR reduces instability in off-policy Q-learning.

Meta-Q-Learning improves meta-Reinforcement Learning using past data.

Improved Q-learning for robot control with faster data-efficiency.

SUNRISE improves off-policy RL algorithms by integrating ensemble methods.

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…

New self-imitation learning method improves performance in continuous control tasks.

Minimal assumptions analysis of Q-learning with time-varying policies.

Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitiv…

Improved sample complexity for target Q-learning in finite MDPs with generative oracle.

Q($Δ$)-Learning improves Q-Learning by separating action-value functions into different time scales.

New insights into experience replay in RL algorithms.

A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.

Continuous-time Q-learning theory developed for reinforcement learning.

Multi-step methods such as Retrace($λ$) and $n$-step $Q$-learning have become a crucial component of modern deep reinforcement learning agents. These methods are often evaluated as a part of bigger architectures and their evaluations rarely include enough samples to draw statistically significant conclusions about thei…

Efficient RL method uses SSL for small labeled data in health outcomes.

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning…

Study evaluates reinforcement learning for trading diverse stocks, finds Q-learning outperforms.

RL enhances LLM planning but introduces spurious solutions and diversity collapse.

Optimized RL algorithms perform well on offline datasets, outperforming fully trained agents.

Paper presents a new method for Bayesian deep learning that scales to Atari games.

In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best …

Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behavio…

Novel RL algorithms learn from human interaction data without exploration.

CQL learns conservative Q-functions to improve offline RL performance.

Proposes Constrained Q-learning for reinforcement learning with constraints.

Enhances SAC for better sample efficiency in continuous-action tasks.

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy pro…

Unified framework for finite-sample RL algorithms using Lyapunov theory.

In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based…

Automates bias control in reinforcement learning algorithms.

It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequ…

Simplifies BCQ to match and outperform state-of-the-art in offline RL benchmarks.

Paper establishes convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.

Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of some variant of Bellman operator that is not necessarily a contraction. As a result, they may easily …

Double Q-learning has the same mean-squared error as Q-learning under certain conditions.

This paper explores a new DRL algorithm that approximates both state-value and state-action functions.

Proposes a value-based method for continuous control without an actor.

This paper benchmarks batch RL algorithms on Atari, finding DQN and partially-trained policies perform best.

Paper analyzes Q-learning with neural networks, proving a fast convergence rate.

DE-QT detects optimal Q-learning stopping points.

This paper analyzes momentum Q-learning with finite-sample guarantees.

EBQL reduces bias in Q-learning for improved performance.

PQ-learning improves Q-learning by periodically updating target estimates.

Linear Q-learning converges to a bounded set without divergence.

Study off-policy evaluation in partially observable environments, reducing bias and errors.

Proposes a new Deep Recurrent Double Q-Learning model for Atari games.

Adapts GRPO for off-policy RL, improving reward.

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.