arXiv research

A new algorithm reduces sample complexity for learning Q-functions in reinforcement learning.

SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.

The paper introduces a new method for estimating optimal policies in dynamic treatment regimes using information geometry.

Study on $Q$-function estimation for continuous state-action MDPs, deriving rates and conditions.

This paper develops q-learning methods for mean-field control problems.

New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.

Q-learning is one of the most popular methods in Reinforcement Learning (RL). Transfer Learning aims to utilize the learned knowledge from source tasks to help new tasks to improve the sample complexity of the new tasks. Considering that data collection in RL is both more time and cost consuming and Q-learning converge…

Improves DRL for long-term causal inference with semiparametric methods.

Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the…

Continuous-time Q-learning theory developed for reinforcement learning.

Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some glo…

This paper explores how IV methods can improve Q-function estimates in offline policy evaluation.

The objective is to study an on-line Hidden Markov model (HMM) estimation-based Q-learning algorithm for partially observable Markov decision process (POMDP) on finite state and action sets. When the full state observation is available, Q-learning finds the optimal action-value function given the current action (Q func…

Proposes a new reinforcement learning algorithm using Q-function.

We present a new Q-function operator for temporal difference (TD) learning methods that explicitly encodes robustness against significant rare events (SRE) in critical domains. The operator, which we call the $κ$-operator, allows to learn a robust policy in a model-based fashion without actually observing the SRE. We i…

The paper proves the convergence of Q-value for Gaussian rewards.

New RL method learns K-step lookahead Q-functions for fixed-horizon MDPs.

We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal $Q$-func…

Proposes a RL method using simulators for stabilizing uncertain systems.

New method estimates and optimizes policy differences using orthogonal learning.

In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based…

New approach transfers rewards learned in one environment to reinforcement learning in a new environment.

DQNs can approximate optimal Q-functions with high accuracy on compact sets.

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…

MACC learns communication protocols by adapting counterfactual reasoning.

Deep Q-Learning models optimal exercise strategies for option-type products.

Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL). It is a necessary component of policy iteration and can be used for variance reduction in policy gradient methods. Therefore its quality has a significant impact on most RL algorithms. Motivated by manifol…

ZDPG learns model-free policies without critics, improving on PG.

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…

SF-DQN improves RL transfer by learning successor features.

A new method reduces complexity in estimating dynamic choice models.

Proposes a value-based method for continuous control without an actor.

Explains agent behavior through intended outcomes in reinforcement learning.

CQL learns conservative Q-functions to improve offline RL performance.

A variety of machine learning models have been proposed to assess the performance of players in professional sports. However, they have only a limited ability to model how player performance depends on the game context. This paper proposes a new approach to capturing game context: we apply Deep Reinforcement Learning (…

New Bellman error estimator improves offline model selection performance.

The paper explores when and why value decomposition algorithms work in cooperative multi-agent reinforcement learning.

Algorithm learns Nash equilibria in stochastic games using entropy-regularized policies.

New algorithms improve robust reinforcement learning under uncertainty.

This work improves sample efficiency in neural function approximation for reinforcement learning.

CriticSMC improves planning efficiency in constrained environments.

We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experimen…

A simple and natural algorithm for reinforcement learning (RL) is Monte Carlo Exploring Starts (MCES), where the Q-function is estimated by averaging the Monte Carlo returns, and the policy is improved by choosing actions that maximize the current estimate of the Q-function. Exploration is performed by "exploring start…

New RL method reduces sample complexity for large state-action spaces.

Online random forests improve Q-learning performance in specific gym environments.

Asynchronous Q-learning achieves optimal Q-function estimation with reduced sample complexity.

GRAC improves reinforcement learning by self-guiding and self-regularizing.

Previous work in hierarchical reinforcement learning has faced a dilemma: either ignore the values of different possible exit states from a subroutine, thereby risking suboptimal behavior, or represent those values explicitly thereby incurring a possibly large representation cost because exit values refer to nonlocal a…