arXiv research

KL-regularized RL from expert demos can lead to slow, unstable learning.

When we implement a portfolio selection methodology under a mean-risk formulation, it is essential to correctly model investors' risk aversion which may be time-dependent, or even state-dependent during the investment procedure. In this paper, we propose a behavior risk aversion model, which is a piecewise linear funct…

Adapts safe policies for exploration in high-risk settings.

SafeMIL learns safer policies by avoiding risky behavior from non-preferred trajectories.

OSIL learns safe policies from unsafe demonstrations.

Methods for learning to search for structured prediction typically imitate a reference policy, with existing theoretical guarantees demonstrating low regret compared to that reference. This is unsatisfactory in many applications where the reference policy is suboptimal and the goal of learning is to improve upon it. Ca…

The paper analyzes the reward improvement of aligned policies in large language models.

Active-GRPO improves molecular optimization by actively deciding when to imitate or self-improve.

We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account…

RFC enhances humanoid control to imitate complex human motions.

The paper explains why estimating a history-dependent policy can reduce MSE in reinforcement learning.

Paper proposes DAC-ML, a cognitive architecture that learns quickly from few episodes.

ORIL learns a reward function from unlabeled data to improve robot learning.

We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over policy behaviors that can in turn be used to lead policy optimization towar…

The paper introduces SuccessProbaMax to optimize policy success probability in online advertising.

New algorithms achieve logarithmic regret in KL-regularized Markov games.

Study minimax off-policy evaluation in multi-armed bandits with known and unknown behavior policies.

Develops neural network framework for risk-reward optimization problems.

GRPO optimizes LLMs with verifiable rewards, amplifying policy success.

We propose a novel approach to train a multi-modal policy from mixed demonstrations without their behavior labels. We develop a method to discover the latent factors of variation in the demonstrations. Specifically, our method is based on the variational autoencoder with a categorical latent variable. The encoder infer…

Identifies optimal base features for zero-shot adaptation in reinforcement learning.

New methods improve LLM preference optimization by intelligently weighting multiple reference models.

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

AIPS improves ranking policy evaluation by adapting to diverse user behavior.

CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.

Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it ta…

Proposes a new framework to optimize portfolios with reduced estimation errors.

Off-policy learning exhibits greater instability when compared to on-policy learning in reinforcement learning (RL). The difference in probability distribution between the target policy ($π$) and the behavior policy (b) is a major cause of instability. High variance also originates from distributional mismatch. The var…

Learning algorithms are enabling robots to solve increasingly challenging real-world tasks. These approaches often rely on demonstrations and reproduce the behavior shown. Unexpected changes in the environment may require using different behaviors to achieve the same effect, for instance to reach and grasp an object in…

This paper improves Q-learning bounds using reference-advantage decomposition.

CQL (ReDS) learns from varied driving behaviors, improving offline RL performance.

This paper investigates how to efficiently transition and update policies, trained initially with demonstrations, using off-policy actor-critic reinforcement learning. It is well-known that techniques based on Learning from Demonstrations, for example behavior cloning, can lead to proficient policies given limited data…

Study multi-agent RL in OTC markets, learning from agents' interactions.

Study strategic dynamic pricing for buyers with unknown manipulation costs.

Deep learning analyzes healthcare provider actions and patient outcomes.

New estimator GMIPS reduces variance in ranking policy evaluation.

We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what …

We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy …

In reinforcement learning, an agent attempts to learn high-performing behaviors through interacting with the environment, such behaviors are often quantified in the form of a reward function. However some aspects of behavior-such as ones which are deemed unsafe and to be avoided-are best captured through constraints. W…

This paper solves steady-state planning for multichain MDPs.

Method learns evolving policies in healthcare contexts.

Study improves off-policy evaluation from non-i.i.d. bandit samples.

DR-RPO optimizes robust policies in RL with limited interaction, achieving sublinear regret.

Proposes Constrained Q-learning for reinforcement learning with constraints.

Q-Distribution Guided Q-Learning corrects overestimation of uncertain OOD actions in offline RL.

The goal of imitation learning is for an apprentice to learn how to behave in a stochastic environment by observing a mentor demonstrating the correct behavior. Accurate prior knowledge about the correct behavior can reduce the need for demonstrations from the mentor. We present a novel approach to encoding prior knowl…

In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that is the same at each state. This can cause batch RL to be overly conservat…

Sharp analysis improves RLHF sample complexity with KL-regularization.