arXiv research

We solve POMDPs by approximating them as finite-state MDPs.

Policy gradient methods achieve linear convergence in simple MDPs.

Paper establishes new lower bounds for MDPs with changing transition kernels.

Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.

DeepAveragers solves offline RL by solving derived MDPs from static data.

Efficiently identifies best policies in tabular MDPs with reduced computational cost.

New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.

Kernel-UCBVI algorithm balances exploration and exploitation in metric state-action spaces.

Improved algorithm detects changes in RL environments with non-stationary MDPs.

New model selects robustly in adversarial reinforcement learning with unknown corruption.

UCB-Advantage learns MDPs with $O(\sqrt{H^2SAT})$ regret.

We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value e…

New method for estimating and optimizing MDPs without stationarity.

Paper identifies reductive MDPs, solving them in polynomial time.

Develops robust MDPs for unknown disturbances with performance guarantees.

New algorithm reduces sample complexity for planning in MDPs.

NVMDP framework tackles non-stationary MDPs with varying discount rates.

DARLING tackles non-stationary RL with guarantees, improving dynamic regret.

This paper is concerned with offline reinforcement learning (RL), which learns using pre-collected data without further exploration. Effective offline RL would be able to accommodate distribution shift and limited data coverage. However, prior algorithms or analyses either suffer from suboptimal sample complexities or …

In classical reinforcement learning, when exploring an environment, agents accept arbitrary short term loss for long term gain. This is infeasible for safety critical applications, such as robotics, where even a single unsafe action may cause system failure. In this paper, we address the problem of safely exploring fin…

Paper bounds PAC RL sample complexity in deterministic MDPs.

While designing the state space of an MDP, it is common to include states that are transient or not reachable by any policy (e.g., in mountain car, the product space of speed and position contains configurations that are not physically reachable). This leads to defining weakly-communicating or multi-chain MDPs. In this…

We introduce SCAL, an algorithm designed to perform efficient exploration-exploitation in any unknown weakly-communicating Markov decision process (MDP) for which an upper bound $c$ on the span of the optimal bias function is known. For an MDP with $S$ states, $A$ actions and $Γ\leq S$ possible next states, we prove a …

Paper establishes first instance-dependent lower bound for PAC reinforcement learning.

RANDPOL uses randomized networks for efficient reinforcement learning in continuous state and action MDPs.

We propose an actor-critic, model-free, and online Reinforcement Learning (RL) framework for continuous-state continuous-action Markov Decision Processes (MDPs) when the reward is highly sparse but encompasses a high-level temporal structure. We represent this temporal structure by a finite-state machine and construct …

New meta-reinforcement learning method improves performance in finite-horizon MDPs.

CycleFQI tackles offline reinforcement learning for cyclic MDPs, mitigating state distribution mismatch.

We address reinforcement learning problems with finite state and action spaces where the underlying MDP has some known structure that could be potentially exploited to minimize the exploration rates of suboptimal (state, action) pairs. For any arbitrary structure, we derive problem-specific regret lower bounds satisfie…

Improved sample complexity for target Q-learning in finite MDPs with generative oracle.

We propose a new complexity measure for Markov decision processes (MDPs), the maximum expected hitting cost (MEHC). This measure tightens the closely related notion of diameter [JOA10] by accounting for the reward structure. We show that this parameter replaces diameter in the upper bound on the optimal value span of a…

We present an algorithm based on the \emph{Optimism in the Face of Uncertainty} (OFU) principle which is able to learn Reinforcement Learning (RL) modeled by Markov decision process (MDP) with finite state-action space efficiently. By evaluating the state-pair difference of the optimal bias function $h^{*}$, the propos…

We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD methods update a few coordinates of the value and policy estimates as a new state transition is obs…

We consider a reinforcement learning (RL) setting in which the agent interacts with a sequence of episodic MDPs. At the start of each episode the agent has access to some side-information or context that determines the dynamics of the MDP for that episode. Our setting is motivated by applications in healthcare where ba…

New RL method learns K-step lookahead Q-functions for fixed-horizon MDPs.

We consider deterministic Markov decision processes (MDPs) and apply max-plus algebra tools to approximate the value iteration algorithm by a smaller-dimensional iteration based on a representation on dictionaries of value functions. The setup naturally leads to novel theoretical results which are simply formulated due…

New RL difficulty shown for discounted settings.

Logarithmic regret for continuous-time reinforcement learning.

New algorithms learn MDPs with better regret bounds using generative sampling.

This paper improves reinforcement learning efficiency for large-scale MDPs.

Q-MMR evaluates policies using reweighted rewards and moment matching.

New method extends low-rank MDPs to continuous action spaces.

A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. \cite{jin2018q} proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP. In this paper, we adapt Q-learning with…

Study minimizes risk in MDPs with spectral measures.

Deep nets solve MDPs without high dimensions.

This work improves Q-learning for average-reward MDPs, reducing sample and communication complexities in federated settings.

We study online learning of finite Markov decision process (MDP) problems when a side information vector is available. The problem is motivated by applications such as clinical trials, recommendation systems, etc. Such applications have an episodic structure, where each episode corresponds to a patient/customer. Our ob…

This paper sets a lower bound for sample complexity in inverse reinforcement learning.