arXiv research

Efficiently plans large MDPs with weak function approximations.

New method for optimistic planning in MDPs using regularization.

Practical reinforcement learning problems are often formulated as constrained Markov decision process (CMDP) problems, in which the agent has to maximize the expected return while satisfying a set of prescribed safety constraints. In this study, we propose a novel simulator-based method to approximately solve a CMDP pr…

New algorithm reduces sample complexity for planning in MDPs.

This paper presents a unifying framework for reinforcement learning and planning.

This paper solves steady-state planning for multichain MDPs.

This work considers the sample and computational complexity of obtaining an $ε$-optimal policy in a discounted Markov Decision Process (MDP), given only access to a generative model. In this work, we study the effectiveness of the most natural plug-in approach to model-based planning: we build the maximum likelihood es…

State-of-the-art efficient model-based Reinforcement Learning (RL) algorithms typically act by iteratively solving empirical models, i.e., by performing \emph{full-planning} on Markov Decision Processes (MDPs) built by the gathered experience. In this paper, we focus on model-based RL in the finite-state finite-horizon…

Survey of integrating planning and learning in model-based reinforcement learning.

The explore{exploit dilemma is one of the central challenges in Reinforcement Learning (RL). Bayesian RL solves the dilemma by providing the agent with information in the form of a prior distribution over environments; however, full Bayesian planning is intractable. Planning with the mean MDP is a common myopic approxi…

New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.

TensorPlan shows an exponential lower bound for planning in MDPs with linearly realizable value functions.

Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.

In reinforcement learning, the discount factor $γ$ controls the agent's effective planning horizon. Traditionally, this parameter was considered part of the MDP; however, as deep reinforcement learning algorithms tend to become unstable when the effective planning horizon is long, recent works refer to $γ$ as a hyper-p…

New algorithm for RL with horizon-free reward-free exploration for linear MDPs.

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…

The paper studies risk-sensitive MDPs with recursive risk measures.

TensorPlan algorithm finds δ-optimal policies with poly$(H,d)$ queries under linearly realizable state-value function.

XLVINs improve deep reinforcement learning by combining self-supervised learning and neural algorithmic reasoning.

This work uses action equivariance to learn structured latent spaces for reinforcement learning.

The paper establishes a nearly-sharp statistical threshold for efficient learning in Latent MDPs with separated components.

New algorithms accelerate value function convergence in MDPs.

This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded ev…

We introduce and analyse two algorithms for exploration-exploitation in discrete and continuous Markov Decision Processes (MDPs) based on exploration bonuses. SCAL$^+$ is a variant of SCAL (Fruit et al., 2018) that performs efficient exploration-exploitation in any unknown weakly-communicating MDP for which an upper bo…

New RL approach tackles non-linear MDPs without linear assumptions.

New algorithm learns optimal policies in strategic MDPs with private types.

New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.

Efficient RL for linear MDPs with unknown transitions.

This work clarifies the role of inference types in planning.

New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.

Paper improves sample complexity for reward-free RL in low-rank MDPs.

TOMA generates abstract graphs for RL, reducing memory and computation costs.

MOReL learns offline RL policies using pessimistic MDPs.

FMDP-BF algorithm improves RL in factored MDPs with exponential regret reduction.

Paper proposes a privacy-preserving RL algorithm for linear MDPs with theoretical guarantees.

New RL algorithm maximizes CVaR in low-rank MDPs with provable efficiency.

Study minimizes risk in MDPs with spectral measures.

New RL approach uses resets to learn complex MDPs efficiently.

RandQL is a new model-free algorithm for MDPs with a novel learning rate randomization approach.

New $L_1$-Coverage objective simplifies exploration in reinforcement learning.

New RL algorithm achieves nearly optimal performance for linear MDPs.

We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal $Q$-func…

This paper proposes a formal approach to online learning and planning for agents operating in a priori unknown, time-varying environments. The proposed method computes the maximally likely model of the environment, given the observations about the environment made by an agent earlier in the system run and assuming know…

Enhances RL with function approximation, improving regret bounds.

Efficiently computes optimal policies for Entropic Risk Measures.

Optimistic algorithm reduces regret in non-stationary linear MDPs.

Safe exploration in RF-RL doesn't increase sample complexity.

Continuous state spaces and stochastic, switching dynamics characterize a number of rich, realworld domains, such as robot navigation across varying terrain. We describe a reinforcementlearning algorithm for learning in these domains and prove for certain environments the algorithm is probably approximately correct wit…