arXiv research

Unified minimax value interval for off-policy evaluation and optimization.

Unified q-learning for mean-field jump-diffusion models with unobservable population distribution.

Unified framework for analyzing pessimism in off-policy learning with regularized importance sampling.

Unified framework for error quantification in off-policy evaluation with distributional shift.

Unified theory for UCB policies in total and max bandit problems.

Unified framework for policy learning using weak supervision.

Unified approach to Merton's portfolio problem using Pontryagin's principles.

This paper unifies two types of statistical methods for estimating treatment effects.

Unified framework for reliable uncertainty quantification in RL.

Unified framework for understanding GRPO as U-statistic.

Unified DICE estimators as regularized Lagrangians for improved off-policy evaluation.

Unified pair trading approach using hierarchical reinforcement learning.

Unified Bayesian framework for efficient off-policy evaluation and learning in large action spaces.

Unified framework for risk-aware policy learning in contextual bandits.

Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over $\mathbb{R}^d$ and in the latter, over a discrete set of actions each of which is parametrized by a continuous parameter. Such problems are natu…

This paper tackles distribution shift in model-based offline RL, proposing a shifts-aware reward method.

Unified framework for policy improvement in RL with benefits in data efficiency and computation.

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our…

Unified LLM-agent with RL improves financial trading performance.

SUNRISE improves off-policy RL algorithms by integrating ensemble methods.

We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In a…

Unified stopping rules ensure accurate policies in contextual learning.

Unified reinforcement learning methods using hybrid inference.

Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper w…

Unified coverage analysis for linear off-policy evaluation in reinforcement learning.

Paper proposes an RL algorithm to ensure policy performance guarantees.

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates across the action when estimating the gradient, instead of relying only on the action in the sampled trajectory. We es…

Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [2], AlphaGo-Zero from [27]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at t…

We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality. We summarize how this duality may be applied to a variety of reinforcement learning (RL) settings, including policy evaluation or optimization, online or offline learning, and discounted or undisco…

This paper presents a novel nonmyopic adaptive Gaussian process planning (GPP) framework endowed with a general class of Lipschitz continuous reward functions that can unify some active learning/sensing and Bayesian optimization criteria and offer practitioners some flexibility to specify their desired choices for defi…

Accelerates policy optimization in RL with optimistic and adaptive updates.

Recent policy optimization approaches (Schulman et al., 2015a; 2017) have achieved substantial empirical successes by constructing new proxy optimization objectives. These proxy objectives allow stable and low variance policy learning, but require small policy updates to ensure that the proxy objective remains an accur…

Beam search is widely used for approximate decoding in structured prediction problems. Models often use a beam at test time but ignore its existence at train time, and therefore do not explicitly learn how to use the beam. We develop an unifying meta-algorithm for learning beam search policies using imitation learning.…

Study designs logging policies to minimize off-policy evaluation error.

We study a security threat to batch reinforcement learning and control where the attacker aims to poison the learned policy. The victim is a reinforcement learner / controller which first estimates the dynamics and the rewards from a batch data set, and then solves for the optimal policy with respect to the estimates. …

A method for a single policy to solve various tasks across diverse agent morphologies.

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates (or sums) across actions when estimating the gradient, instead of relying only on the action in the sampled trajectory…

POLAR learns efficient data acquisition policies using pretrained belief representations.

Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework…

This paper unifies three regularization methods in batch reinforcement learning.

Unified definition of hallucinations in language models.

A new algorithm CAP learns optimal policies from observational data with confounding bias and missing observations.

Unified framework for stable RL learning with theoretical guarantees.

Entropy regularized algorithms such as Soft Q-learning and Soft Actor-Critic, recently showed state-of-the-art performance on a number of challenging reinforcement learning (RL) tasks. The regularized formulation modifies the standard RL objective and thus generally converges to a policy different from the optimal gree…

RRPI improves offline RL by optimizing policies against worst-case dynamics.

Study investigates key design choices in on-policy RL algorithms.

Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific risk-measures, such as the variance or conditional value at risk (CVaR). In this work, we extend the p…

Unified framework for solving MDPs with stochastic mirror descent.