arXiv research

Study on $Q$-function estimation for continuous state-action MDPs, deriving rates and conditions.

A new algorithm reduces sample complexity for learning Q-functions in reinforcement learning.

Improves DRL for long-term causal inference with semiparametric methods.

The paper introduces a new method for estimating optimal policies in dynamic treatment regimes using information geometry.

New algorithms estimate Q-functions under partial coverage and realizability, improving offline RL guarantees.

The objective is to study an on-line Hidden Markov model (HMM) estimation-based Q-learning algorithm for partially observable Markov decision process (POMDP) on finite state and action sets. When the full state observation is available, Q-learning finds the optimal action-value function given the current action (Q func…

This paper explores how IV methods can improve Q-function estimates in offline policy evaluation.

New method estimates and optimizes policy differences using orthogonal learning.

Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the…

New RL method learns K-step lookahead Q-functions for fixed-horizon MDPs.

A new method reduces complexity in estimating dynamic choice models.

Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some glo…

New approach transfers rewards learned in one environment to reinforcement learning in a new environment.

SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.

This paper develops q-learning methods for mean-field control problems.

ZDPG learns model-free policies without critics, improving on PG.

New Bellman error estimator improves offline model selection performance.

Proposes a value-based method for continuous control without an actor.

New algorithms improve robust reinforcement learning under uncertainty.

Q-learning is one of the most popular methods in Reinforcement Learning (RL). Transfer Learning aims to utilize the learned knowledge from source tasks to help new tasks to improve the sample complexity of the new tasks. Considering that data collection in RL is both more time and cost consuming and Q-learning converge…

Continuous-time Q-learning theory developed for reinforcement learning.

A simple and natural algorithm for reinforcement learning (RL) is Monte Carlo Exploring Starts (MCES), where the Q-function is estimated by averaging the Monte Carlo returns, and the policy is improved by choosing actions that maximize the current estimate of the Q-function. Exploration is performed by "exploring start…

Deep neural networks can estimate Q-values efficiently on low-dimensional state-action spaces.

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

Asynchronous Q-learning achieves optimal Q-function estimation with reduced sample complexity.

Proposes a new reinforcement learning algorithm using Q-function.

We present a new Q-function operator for temporal difference (TD) learning methods that explicitly encodes robustness against significant rare events (SRE) in critical domains. The operator, which we call the $κ$-operator, allows to learn a robust policy in a model-based fashion without actually observing the SRE. We i…

The paper proves the convergence of Q-value for Gaussian rewards.

The paper explores when and why value decomposition algorithms work in cooperative multi-agent reinforcement learning.

We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal $Q$-func…

Proposes a RL method using simulators for stabilizing uncertain systems.

Q-learning requires more samples than minimax bounds suggest for optimal Q-function approximation.

We consider model-free reinforcement learning for infinite-horizon discounted Markov Decision Processes (MDPs) with a continuous state space and unknown transition kernel, when only a single sample path under an arbitrary policy of the system is available. We consider the Nearest Neighbor Q-Learning (NNQL) algorithm to…

In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based…

DQNs can approximate optimal Q-functions with high accuracy on compact sets.

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…

MACC learns communication protocols by adapting counterfactual reasoning.

Boosts Q-learning by using value function bounds.

Deep Q-Learning models optimal exercise strategies for option-type products.

Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. We consider for the first time the semiparametric efficiency limits of OPE in Markov decision processes (MDPs), where actions, rewar…

Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL). It is a necessary component of policy iteration and can be used for variance reduction in policy gradient methods. Therefore its quality has a significant impact on most RL algorithms. Motivated by manifol…

CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…

Enhances RL by controlling policy stochasticity through trajectory entropy constraints.

Off-policy evaluation (OPE) in both contextual bandits and reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. The problem's importance has attracted many proposed solutions, including importance sampling (IS), self…

SF-DQN improves RL transfer by learning successor features.

PQR estimates reward functions from actions and states without assuming state-only rewards.

We introduce and analyze a form of variance-reduced $Q$-learning. For $γ$-discounted MDPs with finite state space $\mathcal{X}$ and action space $\mathcal{U}$, we prove that it yields an $ε$-accurate estimate of the optimal $Q$-function in the $\ell_\infty$-norm using $\mathcal{O} \left(\left(\frac{D}{ ε^2 (1-γ)^3} \ri…