arXiv research

DSPI connects natural policy gradient to policy iteration, proving global convergence.

Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.

Policy iteration is a family of algorithms that are used to find an optimal policy for a given Markov Decision Problem (MDP). Simple Policy iteration (SPI) is a type of policy iteration where the strategy is to change the policy at exactly one improvable state at every step. Melekopoglou and Condon [1990] showed an exp…

Adaptive optimal control of nonlinear dynamic systems with deterministic and known dynamics under a known undiscounted infinite-horizon cost function is investigated. Policy iteration scheme initiated using a stabilizing initial control is analyzed in solving the problem. The convergence of the iterations and the optim…

One-step policy improvement outperforms iterative RL methods on D4RL.

RRPI improves offline RL by optimizing policies against worst-case dynamics.

We consider the infinite-horizon discounted optimal control problem formalized by Markov Decision Processes. We focus on several approximate variations of the Policy Iteration algorithm: Approximate Policy Iteration, Conservative Policy Iteration (CPI), a natural adaptation of the Policy Search by Dynamic Programming a…

Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…

The paper proposes an iterative approach to batch reinforcement learning for safer and more informative data collection.

We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves…

A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or $Q$-function may fail to improve performance---or worse, actually cause the policy performance …

Policy gradient method proves convergence in imperfect-information games.

NeuPL learns diverse policies in strategy games efficiently.

Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [2], AlphaGo-Zero from [27]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at t…

The paper tackles robust reinforcement learning with performance guarantees.

New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.

Entropy regularized algorithms such as Soft Q-learning and Soft Actor-Critic, recently showed state-of-the-art performance on a number of challenging reinforcement learning (RL) tasks. The regularized formulation modifies the standard RL objective and thus generally converges to a policy different from the optimal gree…

In this paper, we prove some convergence results of a special case of optimistic policy iteration algorithm for stochastic shortest path problem. We consider both Monte Carlo and $TD(λ)$ methods for the policy evaluation step under the condition that the termination state will eventually be reached almost surely.

Lower bounds for PI on multi-action MDPs are established, showing complexity grows with action count.

Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally …

Policy gradient methods achieve linear convergence in simple MDPs.

In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation erro…

Efficient local planning with linear approximations for agents with limited simulator access.

New algorithms solve robust MDPs efficiently, significantly faster than existing methods.

MVPI framework optimizes risk in reinforcement learning, improving performance in robot simulations.

Policy-GNN optimizes GNN aggregation for diverse node iterations.

We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiab…

Paper achieves $ε^{-2}$ sample complexity for actor-critic methods with minimal assumptions.

Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL). It is a necessary component of policy iteration and can be used for variance reduction in policy gradient methods. Therefore its quality has a significant impact on most RL algorithms. Motivated by manifol…

Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes that generalizes these approaches in two directions: we consider a larger class of regularizers, and…

Proposes a method for obtaining interval bounds in off-policy evaluation.

Improved API to achieve optimal error bound and query complexity in local planning.

KL-constrained API shows optimization issues and improved with regularization.

Novel algorithm for Markov decision processes using rank-one approximation.

This paper improves self-play learning in games by manipulating experience distributions.

Improved Politex algorithm reduces regret bound to O(√T) with experience replay.

Develops a reinforcement learning algorithm for learning deterministic equilibrium policies in time-inconsistent control problems.

Accelerates policy optimization in RL with optimistic and adaptive updates.

DAC enhances exploration in reinforcement learning with entropy regularization.

Paper tackles offline SSP with value iteration for policy evaluation and learning.

Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.

We propose a method for efficient training of Q-functions for continuous-state Markov Decision Processes (MDPs) such that the traces of the resulting policies satisfy a given Linear Temporal Logic (LTL) property. LTL, a modal logic, can express a wide range of time-dependent logical properties (including "safety") that…

New algorithm minimizes worst-case regret in uncertain, time-varying dynamics.

Monte Carlo Tree Search (MCTS) algorithms perform simulation-based search to improve policies online. During search, the simulation policy is adapted to explore the most promising lines of play. MCTS has been used by state-of-the-art programs for many problems, however a disadvantage to MCTS is that it estimates the va…

Mixed RL improves RL efficiency with dual representations.

We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraints, normalizing flows policy generates samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps av…

VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.

Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theor…