arXiv research

Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.

Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy. In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces. It is b…

Proposes method to learn state abstractions that generalize across environments.

This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.

Paper analyzes history-based RL methods for MDPs, introduces a theoretical framework and practical algorithm.

Paper learns meaningful state and action representations from MDP trajectories.

The paper introduces a method to learn Markov state abstractions for reinforcement learning.

This paper optimizes MDP policies for efficient state aggregation.

TOMA generates abstract graphs for RL, reducing memory and computation costs.

This work uses action equivariance to learn structured latent spaces for reinforcement learning.

This paper simplifies OPE in large state spaces using state abstractions.

This paper presents a way of solving Markov Decision Processes that combines state abstraction and temporal abstraction. Specifically, we combine state aggregation with the options framework and demonstrate that they work well together and indeed it is only after one combines the two that the full benefit of each is re…

We propose a method for efficient training of Q-functions for continuous-state Markov Decision Processes (MDPs) such that the traces of the resulting policies satisfy a given Linear Temporal Logic (LTL) property. LTL, a modal logic, can express a wide range of time-dependent logical properties (including "safety") that…

A new method for solving complex sequential decision-making problems by decomposing them into multiple levels.

Optimal strategy for reinforcement learning with delayed observations.

Many problems in sequential decision making and stochastic control often have natural multiscale structure: sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure, particularly beyond a single level of abstraction, has remained a longstanding challen…

\begin{abstract} We model individual T2DM patient blood glucose level (BGL) by stochastic process with discrete number of states mainly but not solely governed by medication regimen (e.g. insulin injections). BGL states change otherwise according to various physiological triggers which render a stochastic, statisticall…

TempLe learns transition templates for efficient multi-task RL.

In this semi-tutorial paper, we first review the information-theoretic approach to account for the computational costs incurred during the search for optimal actions in a sequential decision-making problem. The traditional (MDP) framework ignores computational limitations while searching for optimal policies, essential…

Survey of integrating planning and learning in model-based reinforcement learning.

Adaptive reduction scheme approximates optimal policy in regularized MDPs.

We study Exo-MDPs to reduce sample complexity in reinforcement learning.

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…

Paper establishes new lower bounds for MDPs with changing transition kernels.

We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present …

New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.

Sequential decision making in the presence of uncertainty and stochastic dynamics gives rise to distributions over state/action trajectories in reinforcement learning (RL) and optimal control problems. This observation has led to a variety of connections between RL and inference in probabilistic graphical models (PGMs)…

Study improves understanding of why agentic theorem provers succeed.

Deep Reinforcement Learning (DRL) is a trending field of research, showing great promise in many challenging problems such as playing Atari, solving Go and controlling robots. While DRL agents perform well in practice we are still missing the tools to analayze their performance and visualize the temporal abstractions t…

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…

Optimizes learning policies in MDPs with weakly communicating structure.

A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…

New method removes oracle and reduces memory usage for robust MDPs.

DeepAveragers solves offline RL by solving derived MDPs from static data.

New RL method learns to skip states in linearly $q^π$-realizable MDPs, simplifying to linear MDPs.

Optimizes learning policies in average-reward MDPs with improved sample complexity.

Efficiently plans large MDPs with weak function approximations.

We solve POMDPs by approximating them as finite-state MDPs.

Reward suffices for convex MDPs, expanding RL to new problems.

Practical reinforcement learning problems are often formulated as constrained Markov decision process (CMDP) problems, in which the agent has to maximize the expected return while satisfying a set of prescribed safety constraints. In this study, we propose a novel simulator-based method to approximately solve a CMDP pr…

The paper addresses statistical estimation in MDPs with confounders using instrumental variables.

Study on regret minimization in deterministic MDPs.

Improved regret bound for MNL MDPs with variance-aware approach.

New method tackles MDPs by learning normalized representations efficiently.

A new parallel algorithm for learning optimal policies in MDPs with low communication costs.

It has recently been shown that if feedback effects of decisions are ignored, then imposing fairness constraints such as demographic parity or equality of opportunity can actually exacerbate unfairness. We propose to address this challenge by modeling feedback effects as Markov decision processes (MDPs). First, we prop…

The paper introduces MDP homomorphic networks for faster reinforcement learning.

UCBVI-γ algorithm minimizes regret in discounted MDPs.