arXiv research

Optimistic NPG improves policy optimization in online RL with efficient sample complexity.

New algorithm reduces regret in sequential decision-making problems.

Accelerates policy optimization in RL with optimistic and adaptive updates.

Develops ODRPO to improve RL algorithms with better performance and stability.

In this paper, we prove some convergence results of a special case of optimistic policy iteration algorithm for stochastic shortest path problem. We consider both Monte Carlo and $TD(λ)$ methods for the policy evaluation step under the condition that the termination state will eventually be reached almost surely.

Optimistic actor-critic tackles linear MDPs with parametric policies.

Optimistic PPO variant solves linear MDPs with improved regret bound.

PROPO tackles non-stationary MDPs with efficient policy optimization.

Proposes H-UCRL for efficient model-based RL with sublinear regret.

New method minimizes regret in AMDP with high probability.

While policy-based reinforcement learning (RL) achieves tremendous successes in practice, it is significantly less understood in theory, especially compared with value-based RL. In particular, it remains elusive how to design a provably efficient policy optimization algorithm that incorporates exploration. To bridge su…

New RL algorithm tackles online robust MDPs with uncertainty.

A new policy for contextual bandits adapts to reward vector shifts.

New batch RL method avoids overly optimistic policies.

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

DORIS algorithm achieves no-regret learning in Markov games with adversarial opponents.

New method for optimistic planning in MDPs using regularization.

Policy optimization methods are one of the most widely used classes of Reinforcement Learning (RL) algorithms. Yet, so far, such methods have been mostly analyzed from an optimization perspective, without addressing the problem of exploration, or by making strong assumptions on the interaction with the environment. In …

Study optimal policy regret in partially observable Markov games with adaptive opponents.

Algorithm converges to Nash equilibria in competitive games.

New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.

Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-p…

Improved POMDP regret to sqrt(T) with known observation model.

We consider the problem of online planning in a Markov Decision Process when given only access to a generative model, restricted to open-loop policies - i.e. sequences of actions - and under budget constraint. In this setting, the Open-Loop Optimistic Planning (OLOP) algorithm enjoys good theoretical guarantees but is …

New algorithms reduce regret in online MDPs by adapting to data and variance.

Paper proposes an RL algorithm to ensure policy performance guarantees.

In many cases an intelligent agent may want to learn how to mimic a single observed demonstrated trajectory. In this work we consider how to perform such procedural learning from observation, which could help to enable agents to better use the enormous set of video data on observation sequences. Our approach exploits t…

New rule reduces exploration regret to logarithmic, improving bad episode handling.

Paper optimizes GAIL for online and offline learning with linear approximations.

Robust Optimization has traditionally taken a pessimistic, or worst-case viewpoint of uncertainty which is motivated by a desire to find sets of optimal policies that maintain feasibility under a variety of operating conditions. In this paper, we explore an optimistic, or best-case view of uncertainty and show that it …

RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for PO with mediator feedback.

Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.

Online Apprenticeship Learning aims to match expert performance without access to cost functions.

Optimistic algorithm reduces regret in non-stationary linear MDPs.

This paper studies the statistical theory of batch data reinforcement learning with function approximation. Consider the off-policy evaluation problem, which is to estimate the cumulative value of a new target policy from logged history generated by unknown behavioral policies. We study a regression-based fitted Q iter…

While learning in an unknown Markov Decision Process (MDP), an agent should trade off exploration to discover new information about the MDP, and exploitation of the current knowledge to maximize the reward. Although the agent will eventually learn a good or optimal policy, there is no guarantee on the quality of the in…

Paper proposes a new dynamic pricing method with always-valid online statistical learning.

Bayesian classifier improves robustness with optimistic score ratio.

We show that the optimistic limits of the colored Jones polynomials of the hyperbolic knots coincide with the optimistic limits of the Kashaev invariants modulo $4π^2$.

The paper analyzes CMDPs, balancing exploration and exploitation to avoid constraint violations.

We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning. Optimistic approaches presented in the literature apply an optimistic boost to the value estimate at each state-action pair and select actions that are greedy with respect to the resulting optimistic value f…

New method selects best offline RL policies from logged data.

New algorithm learns optimal policy with multi-step lookahead information.

The optimistic limit is the mathematical formulation of the classical limit which is a physical method to expect the actual limit by using saddle point method of certain potential function. The original optimistic limit of the Kashaev invariant was formulated by Yokota, and a modified formulation was suggested by the a…

Optimistic estimate predicts best fitting performance of nonlinear models.

New principle optimizes bandit decisions with context.

New RL algorithm learns good actions from offline data, reducing uncertainty and divergence.

Real-world applications require RL algorithms to act safely. During learning process, it is likely that the agent executes sub-optimal actions that may lead to unsafe/poor states of the system. Exploration is particularly brittle in high-dimensional state/action space due to increased number of low-performing actions. …