arXiv research

Policy mirror ascent achieves Nash equilibrium in mean field games without a population generative model.

MoMA improves model-based RL by using unrestricted policy classes.

A new framework improves reinforcement learning algorithms with policy guarantees.

New algorithm for optimizing statistical utilities in bandits.

We present a new algorithm based on an gradient ascent for a general Active Exploration bandit problem in the fixed confidence setting. This problem encompasses several well studied problems such that the Best Arm Identification or Thresholding Bandits. It consists of a new sampling rule based on an online lazy mirror …

This paper introduces a new method to evaluate multiple policies simultaneously.

Method identifies mixed Nash equilibria in high dimensions for training mixtures of GANs.

New mirror maps improve PMD performance in reinforcement learning.

The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The …

Mirror descent method improved RL algorithms.

Novel framework for policy optimization with general parameterization and linear convergence.

SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.

DDO-RM improves reward-based policies by converting reward scores into a target distribution.

OMWU shows last iterate convergence in convex-concave games.

GPMD solves regularized RL with linear convergence, promoting structural policies.

New framework for PMD convergence in non-tabular environments.

We accelerate PMD algorithms for reinforcement learning using functional methods.

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency.…

Algorithm converges to Nash equilibria in competitive games.

Unified framework for solving MDPs with stochastic mirror descent.

Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…

Enhances PMD with lookahead to improve RL performance.

Revel tackles safe exploration in RL with verified symbolic policies.

NPMD uses CNNs to optimize policies on low-dimensional manifolds, reducing sample complexity.

Paper proposes MUCS for more reliable TDA in diffusion models.

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are com…

New RL method nearly optimally learns policies with generative models.

Risk management in dynamic decision problems is a primary concern in many fields, including financial investment, autonomous driving, and healthcare. The mean-variance function is one of the most widely used objective functions in risk management due to its simplicity and interpretability. Existing algorithms for mean-…

This paper extends Mirror Descent to Riemannian manifolds for optimization.

Adaptive OMD reduces variance in learning optimal strategies for imperfect information games.

DORIS algorithm achieves no-regret learning in Markov games with adversarial opponents.

Dynamic Programming (DP) provides standard algorithms to solve Markov Decision Processes. However, these algorithms generally do not optimize a scalar objective function. In this paper, we draw connections between DP and (constrained) convex optimization. Specifically, we show clear links in the algorithmic structure b…

A new parallel algorithm for learning optimal policies in MDPs with low communication costs.

Uniform sampling of training data has been commonly used in traditional stochastic optimization algorithms such as Proximal Stochastic Gradient Descent (prox-SGD) and Proximal Stochastic Dual Coordinate Ascent (prox-SDCA). Although uniform sampling can guarantee that the sampled stochastic quantity is an unbiased estim…

The paper interprets policy-gradient algorithms using continuation theory.

Algorithm optimizes system design and control for better rewards.

This work finds mixed equilibria in machine learning problems using measures and simultaneous gradient ascent-descent.

AIF reformulated as convex MDP for adaptive behavior.

A new RL algorithm POWR learns world models to estimate action-values.

Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…

This paper balances bias and variance in adaptive importance sampling using mirror descent.

Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradi…

Paper proposes a policy gradient method for confounded POMDPs.

This paper studies GAIL's global convergence for general MDP and nonlinear rewards.

New algorithm for reinforcement learning reduces complexity and guarantees convergence.

Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theor…

Improved reinforcement learning algorithm with linear approximation for unknown dynamics.

A new approach to fine-tuning LLMs with human feedback.