arXiv research

A new framework improves reinforcement learning algorithms with policy guarantees.

Develops methods to create consistent surrogate models for agent-based simulators.

When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a colle…

PPOS improves PPO by smoothing the surrogate objective function.

Adapts GRPO for off-policy RL, improving reward.

The paper studies consistency of surrogate loss procedures under constrained classifiers.

Robust optimization and statistical robustness improve robot navigation policies.

Revisits PPO design choices, exposing failure modes and proposing alternatives.

Learning optimal resource allocation policies in wireless systems can be effectively achieved by formulating finite dimensional constrained programs which depend on system configuration, as well as the adopted learning parameterization. The interest here is in cases where system models are unavailable, prompting method…

We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonconvex functions. For such a problem, we construct a sequence of surrogate convex constrained optimiza…

Recent policy optimization approaches have achieved substantial empirical success by constructing surrogate optimization objectives. The Approximate Policy Iteration objective (Schulman et al., 2015a; Kakade and Langford, 2002) has become a standard optimization target for reinforcement learning problems. Using this ob…

Efficient policy learning from observational data using weighted classification reductions.

RRPI improves offline RL by optimizing policies against worst-case dynamics.

Seq2Seq models speed up epidemic model predictions.

New method improves solving combinatorial optimization problems with smoothed policies.

In this paper, we point out a fundamental property of the objective in reinforcement learning, with which we can reformulate the policy gradient objective into a perceptron-like loss function, removing the need to distinguish between on and off policy training. Namely, we posit that it is sufficient to only update a po…

Multi-step greedy policies have been extensively used in model-based reinforcement learning (RL), both when a model of the environment is available (e.g.,~in the game of Go) and when it is learned. In this paper, we explore their benefits in model-free RL, when employed using multi-step dynamic programming algorithms: …

CJE calibrates cheap LLM judges against an oracle, achieving high accuracy at a fraction of the cost.

Contextual policy search (CPS) is a class of multi-task reinforcement learning algorithms that is particularly useful for robotic applications. A recent state-of-the-art method is Contextual Covariance Matrix Adaptation Evolution Strategies (C-CMA-ES). It is based on the standard black-box optimization algorithm CMA-ES…

Managers of US National Forests must decide what policy to apply for dealing with lightning-caused wildfires. Conflicts among stakeholders (e.g., timber companies, home owners, and wildlife biologists) have often led to spirited political debates and even violent eco-terrorism. One way to transform these conflicts into…

PPO's gradients are heavy-tailed, affecting learning; a robust estimator improves performance.

We use surrogate losses to obtain several new regret bounds and new algorithms for contextual bandit learning. Using the ramp loss, we derive new margin-based regret bounds in terms of standard sequential complexity measures of a benchmark class of real-valued regression functions. Using the hinge loss, we derive an ef…

The paper interprets policy-gradient algorithms using continuation theory.

Policy optimization is a core component of reinforcement learning (RL), and most existing RL methods directly optimize parameters of a policy based on maximizing the expected total reward, or its surrogate. Though often achieving encouraging empirical success, its underlying mathematical principle on {\em policy-distri…

The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.

Estimating the long-term effects of treatments is of interest in many fields. A common challenge in estimating such treatment effects is that long-term outcomes are unobserved in the time frame needed to make policy decisions. One approach to overcome this missing data problem is to analyze treatments effects on an int…

Accelerates policy optimization in RL with optimistic and adaptive updates.

New PG losses improve decision optimization in misspecified models.

Estimates long-term effects of new treatments using historical and short-term data.

TRM improves long-horizon LLM RL by masking divergent sequences.

Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-triv…

TRM improves long-horizon reinforcement learning for LLMs by masking divergent sequences.

Bayesian approach uses Gaussian process for reinforcement learning.

Efficient RL method uses SSL for small labeled data in health outcomes.

This work extends ME-RL using diffusion models to sample optimal policies.

A method to improve surrogate model accuracy using multiple fidelity models.

POLAR learns efficient data acquisition policies using pretrained belief representations.

Proposes a new sampling policy for ranking and selection problems.

What is a good exploration strategy for an agent that interacts with an environment in the absence of external rewards? Ideally, we would like to get a policy driving towards a uniform state-action visitation (highly exploring) in a minimum number of steps (fast mixing), in order to ease efficient learning of any goal-…

Study finds dividend payout policy positively impacts firm profitability.

We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our result…

Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves da…

TAMO optimizes multiple objectives in-context using transformers.

New algorithm uses machine learning to predict rewards for decision-making problems.

Comment on entropy learning for dynamic treatment regimes.

Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved. Nevertheless, TRPO has been considered a heuristic algorithm inspi…

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the wo…

The Exploration-Exploitation tradeoff arises in Reinforcement Learning when one cannot tell if a policy is optimal. Then, there is a constant need to explore new actions instead of exploiting past experience. In practice, it is common to resolve the tradeoff by using a fixed exploration mechanism, such as $ε$-greedy ex…