arXiv research

The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.

The paper tackles reinforcement learning with exogenous variables and rewards.

A new estimator for state values in reinforcement learning reduces complexity and improves convergence.

Extends linear MDP to handle nonlinear rewards.

New Q-learning method achieves optimal sample complexity for average-reward problems.

The paper studies reward concentration in MDPs, covering asymptotic and non-asymptotic settings.

ARL uses queries to learn rewards, focusing on cost vs. reward value.

Exogenous state variables and rewards can slow down reinforcement learning by injecting uncontrolled variation into the reward signal. We formalize exogenous state variables and rewards and identify conditions under which an MDP with exogenous state can be decomposed into an exogenous Markov Reward Process involving on…

We propose a new complexity measure for Markov decision processes (MDPs), the maximum expected hitting cost (MEHC). This measure tightens the closely related notion of diameter [JOA10] by accounting for the reward structure. We show that this parameter replaces diameter in the upper bound on the optimal value span of a…

Novel framework for risk-sensitive reinforcement learning using martingale decomposition.

This work extends reinforcement learning to handle non-cumulative objectives.

Reward suffices for convex MDPs, expanding RL to new problems.

Current reinforcement learning methods fail if the reward function is imperfect, i.e. if the agent observes reward different from what it actually receives. We study this problem within the formalism of Corrupt Reward Markov Decision Processes (CRMDPs). We show that if the reward corruption in a CRMDP is sufficiently "…

Logarithmic regret for continuous-time reinforcement learning.

UCRL-CMDP algorithm optimizes RL with constraints on average costs.

New RL method explores environments without rewards, achieving efficient policy generation.

New algorithm learns optimal policy for average reward MDPs with sample complexity matching lower bound.

We consider reinforcement learning in changing Markov Decision Processes where both the state-transition probabilities and the reward functions may vary over time. For this problem setting, we propose an algorithm using a sliding window approach and provide performance guarantees for the regret evaluated against the op…

AIF reformulated as convex MDP for adaptive behavior.

We study a multi-armed bandit problem where the rewards exhibit regime switching. Specifically, the distributions of the random rewards generated from all arms are modulated by a common underlying state modeled as a finite-state Markov chain. The agent does not observe the underlying state and has to learn the transiti…

Develops first-order methods for average-reward MDPs with strong guarantees.

The paper introduces a method to learn Markov state abstractions for reinforcement learning.

Model-free reinforcement learning is known to be memory and computation efficient and more amendable to large scale problems. In this paper, two model-free algorithms are introduced for learning infinite-horizon average-reward Markov Decision Processes (MDPs). The first algorithm reduces the problem to the discounted-r…

We characterize value functions in partially observable MDPs as semi-algebraic sets.

New algorithm achieves optimal regret in average reward MDPs without prior bias information.

Modeling decision-making dynamics in MDD patients using RL-HMM.

Efficient algorithm for learning from indirect feedback in complex decision-making scenarios.

New algorithm for reward-free RL with linear function approximation, reducing sample complexity.

Faster algorithms for solving multichain MDPs under average-reward criterion.

Studies on massive open online courses (MOOCs) users discuss the existence of typical profiles and their impact on the learning process of the students. However defining the typical behaviors as well as classifying the users accordingly is a difficult task. In this paper we suggest two methods to model MOOC users behav…

This paper presents four different ways of looking at the well-known Least Squares Temporal Differences (LSTD) algorithm for computing the value function of a Markov Reward Process, each of them leading to different insights: the operator-theory approach via the Galerkin method, the statistical approach via instrumenta…

Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.

Paper presents an algorithm for optimal regret in communicating Markov decision processes.

Variance plays a crucial role in risk-sensitive reinforcement learning, and most risk measures can be analyzed via variance. In this paper, we consider two law-invariant risks as examples: mean-variance risk and exponential utility risk. With the aid of the state-augmentation transformation (SAT), we show that, the two…

New RL method tackles dynamic MDPs with evolving rewards and states.

We consider Markov Decision Processes (MDPs) where the rewards are unknown and may change in an adversarial manner. We provide an algorithm that achieves state-of-the-art regret bound of $O( \sqrt{τ(\ln|S|+\ln|A|)T}\ln(T))$, where $S$ is the state space, $A$ is the action space, $τ$ is the mixing time of the MDP, and $…

This work improves Q-learning for average-reward MDPs, reducing sample and communication complexities in federated settings.

Study optimal offline RL with uncertainty sets and distribution shifts.

TLRS improves predictive power of mined formulaic alpha factors.

In reinforcement learning (RL) , one of the key components is policy evaluation, which aims to estimate the value function (i.e., expected long-term accumulated reward) of a policy. With a good policy evaluation method, the RL algorithms will estimate the value function more accurately and find a better policy. When th…

We consider online learning for minimizing regret in unknown, episodic Markov decision processes (MDPs) with continuous states and actions. We develop variants of the UCRL and posterior sampling algorithms that employ nonparametric Gaussian process priors to generalize across the state and action spaces. When the trans…

Finding optimal policies which maximize long term rewards of Markov Decision Processes requires the use of dynamic programming and backward induction to solve the Bellman optimality equation. However, many real-world problems require optimization of an objective that is non-linear in cumulative rewards for which dynami…

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology, behavioral economics, an…

We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance p…

A single policy suffices for near-optimal parallel exploration in RL.

We consider a planning problem where the dynamics and rewards of the environment depend on a hidden static parameter referred to as the context. The objective is to learn a strategy that maximizes the accumulated reward across all contexts. The new model, called Contextual Markov Decision Process (CMDP), can model a cu…

New algorithm learns policies without explicit rewards for MDPs.

New method reduces sample complexity for robust reinforcement learning.