arXiv research

Optimistic NPG improves policy optimization in online RL with efficient sample complexity.

In this paper, we prove some convergence results of a special case of optimistic policy iteration algorithm for stochastic shortest path problem. We consider both Monte Carlo and $TD(λ)$ methods for the policy evaluation step under the condition that the termination state will eventually be reached almost surely.

PROPO tackles non-stationary MDPs with efficient policy optimization.

New algorithm reduces regret in sequential decision-making problems.

Accelerates policy optimization in RL with optimistic and adaptive updates.

New batch RL method avoids overly optimistic policies.

Develops ODRPO to improve RL algorithms with better performance and stability.

Optimistic actor-critic tackles linear MDPs with parametric policies.

Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.

DORIS algorithm achieves no-regret learning in Markov games with adversarial opponents.

This paper studies the statistical theory of batch data reinforcement learning with function approximation. Consider the off-policy evaluation problem, which is to estimate the cumulative value of a new target policy from logged history generated by unknown behavioral policies. We study a regression-based fitted Q iter…

Optimistic PPO variant solves linear MDPs with improved regret bound.

Proposes H-UCRL for efficient model-based RL with sublinear regret.

New method minimizes regret in AMDP with high probability.

While policy-based reinforcement learning (RL) achieves tremendous successes in practice, it is significantly less understood in theory, especially compared with value-based RL. In particular, it remains elusive how to design a provably efficient policy optimization algorithm that incorporates exploration. To bridge su…

New RL algorithm tackles online robust MDPs with uncertainty.

A new policy for contextual bandits adapts to reward vector shifts.

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

New method selects best offline RL policies from logged data.

New method for optimistic planning in MDPs using regularization.

New method for robust policy evaluation in offline reinforcement learning with sequentially exogenous unobserved confounders.

Policy optimization methods are one of the most widely used classes of Reinforcement Learning (RL) algorithms. Yet, so far, such methods have been mostly analyzed from an optimization perspective, without addressing the problem of exploration, or by making strong assumptions on the interaction with the environment. In …

Study optimal policy regret in partially observable Markov games with adaptive opponents.

Algorithm converges to Nash equilibria in competitive games.

A fundamental problem arising in many areas of machine learning is the evaluation of the likelihood of a given observation under different nominal distributions. Frequently, these nominal distributions are themselves estimated from data, which makes them susceptible to estimation errors. We thus propose to replace each…

Paper proposes a new dynamic pricing method with always-valid online statistical learning.

Optimistic estimate predicts best fitting performance of nonlinear models.

Data-driven decision-making often overestimates benefits due to the winner's curse.

The optimistic limit is the mathematical formulation of the classical limit which is a physical method to expect the actual limit by using saddle point method of certain potential function. The original optimistic limit of the Kashaev invariant was formulated by Yokota, and a modified formulation was suggested by the a…

New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.

Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-p…

Improved POMDP regret to sqrt(T) with known observation model.

We consider the problem of online planning in a Markov Decision Process when given only access to a generative model, restricted to open-loop policies - i.e. sequences of actions - and under budget constraint. In this setting, the Open-Loop Optimistic Planning (OLOP) algorithm enjoys good theoretical guarantees but is …

New algorithms reduce regret in online MDPs by adapting to data and variance.

Decouples critic chunk length from policy to improve policy reactivity and performance.

A framework for navigating environments with spatially correlated obstacles and uncertain blockage status.

New method boosts skill learning by encouraging optimistic exploration.

Paper proposes an RL algorithm to ensure policy performance guarantees.

In many cases an intelligent agent may want to learn how to mimic a single observed demonstrated trajectory. In this work we consider how to perform such procedural learning from observation, which could help to enable agents to better use the enormous set of video data on observation sequences. Our approach exploits t…

Optimistic method adapted for faster convex-concave min-max problems.

Optimistic search speeds up change point detection in large datasets.

New rule reduces exploration regret to logarithmic, improving bad episode handling.

Paper optimizes GAIL for online and offline learning with linear approximations.

Robust Optimization has traditionally taken a pessimistic, or worst-case viewpoint of uncertainty which is motivated by a desire to find sets of optimal policies that maintain feasibility under a variety of operating conditions. In this paper, we explore an optimistic, or best-case view of uncertainty and show that it …

The likelihood function is a fundamental component in Bayesian statistics. However, evaluating the likelihood of an observation is computationally intractable in many applications. In this paper, we propose a non-parametric approximation of the likelihood that identifies a probability measure which lies in the neighbor…

RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for PO with mediator feedback.

Online Apprenticeship Learning aims to match expert performance without access to cost functions.

StoSOO optimistically maximizes noisy, locally smooth functions.