arXiv research

New method recovers diverse policies from expert data using state-action pair weighting.

New findings reveal discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.

This work tackles model-based RL by optimizing state-action queries to learn policies with minimal data.

This paper unifies three regularization methods in batch reinforcement learning.

A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.

GIM learns by analogy to improve reinforcement learning efficiency.

Study optimal offline RL with uncertainty sets and distribution shifts.

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights…

We consider the problem of learning a policy for a Markov decision process consistent with data captured on the state-actions pairs followed by the policy. We assume that the policy belongs to a class of parameterized policies which are defined using features associated with the state-action pairs. The features are kno…

The paper improves Q-learning by incorporating pessimism for better sample efficiency.

Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.

In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correct…

This work compares human feedback methods for reward learning in bandits.

In this paper, we settle the sampling complexity of solving discounted two-player turn-based zero-sum stochastic games up to polylogarithmic factors. Given a stochastic game with discount factor $γ\in(0,1)$ we provide an algorithm that computes an $ε$-optimal strategy with high-probability given $\tilde{O}((1 - γ)^{-3}…

Paper tackles covariate shift in offline IL using less proficient behavior data.

Model-free learning for multi-agent stochastic games is an active area of research. Existing reinforcement learning algorithms, however, are often restricted to zero-sum games, and are applicable only in small state-action spaces or other simplified settings. Here, we develop a new data efficient Deep-Q-learning method…

We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the tas…

Unified framework for solving MDPs with stochastic mirror descent.

Develops a dynamic mean field theory for reinforcement learning.

Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.

New algorithm reduces sample complexity for new tasks by leveraging prior knowledge.

We seek to learn an effective policy for a Markov Decision Process (MDP) with continuous states via Q-Learning. Given a set of basis functions over state action pairs we search for a corresponding set of linear weights that minimizes the mean Bellman residual. Our algorithm uses a Kalman filter model to estimate those …

Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.

We formulate simple assumptions, implying the Robbins-Monro conditions for the $Q$-learning algorithm with the local learning rate, depending on the number of visits of a particular state-action pair (local clock) and the number of iteration (global clock). It is assumed that the Markov decision process is communicatin…

Paper improves RL algorithms with known optimal Q-value predictions.

State-only imitation learning improves dexterous manipulation learning from videos.

Reinforcement learning with trajectory feedback instead of state-action rewards.

In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Buil…

Leveraging an equivalence property in the state-space of a Markov Decision Process (MDP) has been investigated in several studies. This paper studies equivalence structure in the reinforcement learning (RL) setup, where transition distributions are no longer assumed to be known. We present a notion of similarity betwee…

New RL method handles large state-action spaces with complex models.

New method for natural policy gradients converges linearly.

A new framework for offline RL improves policy flexibility and regularity.

New method improves deep policy gradient algorithms by learning relative state values.

A new RL paradigm reduces state-action-value function approximation inefficiency.

New RL method reduces sample complexity for large state-action spaces.

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the…

MetaCURL tackles non-stationary MDPs with optimal dynamic regret.

IDAC improves reinforcement learning efficiency by modeling implicit distributions.

Novel framework proves fast RL convergence in continuous spaces.

New method reduces regret for sparse adversarial SSP problems.

Failures are challenging for learning to control physical systems since they risk damage, time-consuming resets, and often provide little gradient information. Adding safety constraints to exploration typically requires a lot of prior knowledge and domain expertise. We present a safety measure which implicitly captures…

Kernel-UCBVI algorithm balances exploration and exploitation in metric state-action spaces.

New method finds exact Pareto front for MO-MDPs efficiently.

New framework for efficient query-based imitation learning.

Reinforcement learning methods require careful design involving a reward function to obtain the desired action policy for a given task. In the absence of hand-crafted reward functions, prior work on the topic has proposed several methods for reward estimation by using expert state trajectories and action pairs. However…

ESAC improves reinforcement learning by lookahead and intuition.

New algorithm reduces sample complexity for planning in MDPs.