arXiv research

New method for estimating and optimizing MDPs without stationarity.

In order to make good decision under uncertainty an agent must learn from observations. To do so, two of the most common frameworks are Contextual Bandits and Markov Decision Processes (MDPs). In this paper, we study whether there exist algorithms for the more general framework (MDP) which automatically provide the bes…

We consider the recently proposed reinforcement learning (RL) framework of Contextual Markov Decision Processes (CMDP), where the agent interacts with a (potentially adversarial) sequence of episodic tabular MDPs. In addition, a context vector determining the MDP parameters is available to the agent at the start of eac…

Paper studies linear CMDPs, improving sample complexity for both models.

Efficient algorithm for learning from indirect feedback in complex decision-making scenarios.

Efficient RL for linear MDPs with unknown transitions.

We consider a reinforcement learning (RL) setting in which the agent interacts with a sequence of episodic MDPs. At the start of each episode the agent has access to some side-information or context that determines the dynamics of the MDP for that episode. Our setting is motivated by applications in healthcare where ba…

We study the sample complexity of model-based reinforcement learning (henceforth RL) in general contextual decision processes that require strategic exploration to find a near-optimal policy. We design new algorithms for RL with a generic model class and analyze their statistical properties. Our algorithms have sample …

We consider the task of Inverse Reinforcement Learning in Contextual Markov Decision Processes (MDPs). In this setting, contexts, which define the reward and transition kernel, are sampled from a distribution. In addition, although the reward is a function of the context, it is not provided to the agent. Instead, the a…

New model selects robustly in adversarial reinforcement learning with unknown corruption.

This paper sets communication complexity bounds for distributed RL.

CB-RL solves complex decision-making problems with contextual information and exogenous events.

New approach turns optimal stationary RL into non-stationary RL without prior knowledge.

Improved Thompson Sampling reduces regret in contextual bandits and reinforcement learning.

Minimax PAC bounds for learning in exogenous contextual MDPs

New algorithm reduces regret in RL with adversarial corruption.

New framework handles dynamic contexts in reinforcement learning.

The paper tackles cooperative RL with function approximation, achieving near-optimal learning with limited communication.

Efficient algorithm for CMDPs reduces to offline density estimation.

Paper develops a new estimator for MDPs' risk functionals with lower variance and bias.

Study off-policy evaluation and learning in dynamic pricing with context.

In this paper, we consider the problem of online learning of Markov decision processes (MDPs) with very large state spaces. Under the assumptions of realizable function approximation and low Bellman ranks, we develop an online learning algorithm that learns the optimal value function while at the same time achieving ve…

Study batch reinforcement learning methods for personalized medical treatments.

New algorithm reduces regret for linear bandits with unknown noise variance.

In crowd labeling, a large amount of unlabeled data instances are outsourced to a crowd of workers. Workers will be paid for each label they provide, but the labeling requester usually has only a limited amount of the budget. Since data instances have different levels of labeling difficulty and workers have different r…

A new offline RL framework unifies imitation learning and vanilla offline RL.

This paper explains why distributional reinforcement learning is better than vanilla RL using small-loss bounds.

Adaptive reduction scheme approximates optimal policy in regularized MDPs.

We study Exo-MDPs to reduce sample complexity in reinforcement learning.

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…

Paper establishes new lower bounds for MDPs with changing transition kernels.

We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present …

New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…

Optimizes learning policies in MDPs with weakly communicating structure.

RL applied to finance tasks, highlighting challenges and future directions.

A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…

New method removes oracle and reduces memory usage for robust MDPs.

DeepAveragers solves offline RL by solving derived MDPs from static data.

New RL method learns to skip states in linearly $q^π$-realizable MDPs, simplifying to linear MDPs.

Optimizes learning policies in average-reward MDPs with improved sample complexity.

Efficiently plans large MDPs with weak function approximations.

We solve POMDPs by approximating them as finite-state MDPs.

Reward suffices for convex MDPs, expanding RL to new problems.

Practical reinforcement learning problems are often formulated as constrained Markov decision process (CMDP) problems, in which the agent has to maximize the expected return while satisfying a set of prescribed safety constraints. In this study, we propose a novel simulator-based method to approximately solve a CMDP pr…

The paper addresses statistical estimation in MDPs with confounders using instrumental variables.

Study on regret minimization in deterministic MDPs.

Improved regret bound for MNL MDPs with variance-aware approach.