arXiv research

Develops first-order methods for average-reward MDPs with strong guarantees.

In reinforcement learning, an agent attempts to learn high-performing behaviors through interacting with the environment, such behaviors are often quantified in the form of a reward function. However some aspects of behavior-such as ones which are deemed unsafe and to be avoided-are best captured through constraints. W…

New RL algorithms learn policies competitive with best in class without assuming optimal policy.

We present a predictor-corrector framework, called PicCoLO, that can transform a first-order model-free reinforcement or imitation learning algorithm into a new hybrid method that leverages predictive models to accelerate policy learning. The new "PicCoLOed" algorithm optimizes a policy by recursively repeating two ste…

CEFOL uses deep learning for dynamic programming with recursive utility.

Taylor expansions improve reinforcement learning policies.

A Relational Markov Decision Process (RMDP) is a first-order representation to express all instances of a single probabilistic planning domain with possibly unbounded number of objects. Early work in RMDPs outputs generalized (instance-independent) first-order policies or value functions as a means to solve all instanc…

The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency.…

This paper achieves first-order regret bounds in reinforcement learning with large state spaces.

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

RO-TD learns sparse value functions efficiently.

A new method solves bilevel optimization problems in competitive Markov games.

This work analyzes actor-critic methods for faster convergence.

In this paper, we study reinforcement learning (RL) algorithms to solve real-world decision problems with the objective of maximizing the long-term reward as well as satisfying cumulative constraints. We propose a novel first-order policy optimization method, Interior-point Policy Optimization (IPO), which augments the…

ZOSPI improves RL policies with global value function exploitation.

As the most successful variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely applied across various domains with several advantages: efficient data utilization, easy implementation, and good parallelism. In this paper, a first-order gradient reinforcemen…

Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves da…

New method prevents RLHF alignment collapse by accounting for policy's influence on reward model updates.

Improves policy optimization with polylog(T) regret bounds for stochastic losses.

In this paper we study a continuous time, optimal stochastic investment problem under limited resources in a market with N firms. The investment processes are subject to a time-dependent stochastic constraint. Rather than using a dynamic programming approach, we exploit the concavity of the profit functional to derive …

The paper explores how different loss functions impact reinforcement learning algorithms.

Study policy gradient and actor-critic methods for continuous-time reinforcement learning.

New method for estimating value of optimal policies in uncertain scenarios.

Paper improves COCO problem, reducing constraint violation at the cost of slightly more regret.

Proximal policy optimization(PPO) has been proposed as a first-order optimization method for reinforcement learning. We should notice that an exterior penalty method is used in it. Often, the minimizers of the exterior penalty functions approach feasibility only in the limits as the penalty parameter grows increasingly…

QAM uses adjoint matching to optimize continuous-action RL policies efficiently.

New algorithms reduce regret in online MDPs by adapting to data and variance.

Optimizes trading strategy considering alpha decay and transaction costs.

Sample efficiency is critical in solving real-world reinforcement learning problems, where agent-environment interactions can be costly. Imitation learning from expert advice has proved to be an effective strategy for reducing the number of interactions required to train a policy. Online imitation learning, which inter…

Second-order estimator improves continuous-time policy evaluation.

Mirror descent method improved RL algorithms.

Novel algorithm for Markov decision processes using rank-one approximation.

New framework optimizes multi-asset portfolio choice for high dimensions.

New method improves stability of soft FQI for offline RL.

Continuous-time Q-learning theory developed for reinforcement learning.

We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of stochastic policy gradient for the realized MDP. In particular, using stoc…

This paper develops q-learning methods for mean-field control problems.

Deep learning solves dynamic programming with recursive utility.

Researchers quantify risk exposure and sensitivities in financial markets under model uncertainty.

Proposes efficient FOBO algorithms for global maxima of expensive functions.

Generative Adversarial Imitation Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies. Different from Reinforcement Learning (RL), GAIL takes advantage of demonstration data by experts (e.g., human), and learns both the policy and reward function of the unknown environme…

In a market with one safe and one risky asset, an investor with a long horizon, constant investment opportunities, and constant relative risk aversion trades with small proportional transaction costs. We derive explicit formulas for the optimal investment policy, its implied welfare, liquidity premium, and trading volu…

Study on sample complexity of policy gradient for stabilizing linear systems under multiplicative noise.

Under appropriate cooperation protocols and parameter choices, fully decentralized solutions for stochastic optimization have been shown to match the performance of centralized solutions and result in linear speedup (in the number of agents) relative to non-cooperative approaches in the strongly-convex setting. More re…

This work overcomes bias in concave multi-objective reinforcement learning.

Improved sampling efficiency with Random Reshuffling for Langevin dynamics.

Proposes a new sampling policy for ranking and selection problems.