arXiv research

We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.

Study model selection in batch policy optimization with three error sources.

The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.

New algorithm learns optimal policy with multi-step lookahead information.

We study a security threat to batch reinforcement learning and control where the attacker aims to poison the learned policy. The victim is a reinforcement learner / controller which first estimates the dynamics and the rewards from a batch data set, and then solves for the optimal policy with respect to the estimates. …

New batch RL method avoids overly optimistic policies.

Study nonparametric contextual bandits with batched updates, achieving optimal regret.

Batched Neural Bandits reduces policy updates in sequential decision-making.

Sparse feature selection improves batch RL efficiency.

Active search is a learning paradigm for actively identifying as many members of a given class as possible. A critical target scenario is high-throughput screening for scientific discovery, such as drug or materials discovery. In this paper, we approach this problem in Bayesian decision framework. We first derive the B…

STEEL tackles batch RL with singularity, improving policy optimization.

In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that is the same at each state. This can cause batch RL to be overly conservat…

We analyze how batch learning impacts bandit problems.

A new algorithm learns policies from batch data in hierarchical RL.

GN algorithm solves batched bandit for nondegenerate functions near-optimally.

In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the re…

We develop an off-policy actor-critic algorithm for learning an optimal policy from a training set composed of data from multiple individuals. This algorithm is developed with a view towards its use in mobile health.

LF-IBIS learns optimal policies online without explicit likelihood.

When learning policies for real-world domains, two important questions arise: (i) how to efficiently use pre-collected off-policy, non-optimal behavior data; and (ii) how to mediate among different competing objectives and constraints. We thus study the problem of batch policy learning under multiple constraints, and o…

Finite-horizon sequential experimental design (SED) arises naturally in many contexts, including hyperparameter tuning in machine learning among more traditional settings. Computing the optimal policy for such problems requires solving Bellman equations, which are generally intractable. Most existing work resorts to se…

When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a colle…

ABS dynamically adjusts batch size based on policy stability, improving RL performance.

Bayesian sOED uses PG reinforcement learning for efficient experiment design.

Critic-regularized regression improves offline RL performance.

The paper proposes an iterative approach to batch reinforcement learning for safer and more informative data collection.

Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.

Paper addresses OPE for dependent bandit samples using MDS and batch updates.

A novel approach learns goal-conditioned policies for locomotion using batch RL.

Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probability that the trained policy performs better than the behavioural policy, also called baseline in this…

Two algorithms achieve optimal regret with limited adaptivity in multinomial logistic bandits.

Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function $V$. Two novel algorithms are proposed to approximate the true value…

Study on adaptivity constraints in linear contextual bandits with optimal design.

Memory-efficient algorithm reduces variance in off-policy RL.

We study the problem of multi-agent reinforcement learning (MARL) with adaptivity constraints -- a new problem motivated by real-world applications where deployments of new policies are costly and the number of policy updates must be minimized. For two-player zero-sum Markov Games, we design a (policy) elimination base…

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free \emph{batch} settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed ex…

Designs a single policy for collecting data to train near-optimal policies.

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learn…

Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-triv…

The design of multiple experiments is commonly undertaken via suboptimal strategies, such as batch (open-loop) design that omits feedback or greedy (myopic) design that does not account for future effects. This paper introduces new strategies for the optimal design of sequential experiments. First, we rigorously formul…

This paper improves sample complexity for AC and NAC algorithms under Markovian sampling.

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this pape…

New algorithm for sequential off-policy learning improves performance over batch methods.

New RL algorithm learns good actions from offline data, reducing uncertainty and divergence.

MOPO optimizes offline RL by penalizing dynamics uncertainty.

Most prior work on active learning of classifiers has focused on sequentially selecting one unlabeled example at a time to be labeled in order to reduce the overall labeling effort. In many scenarios, however, it is desirable to label an entire batch of examples at once, for example, when labels can be acquired in para…

Paper proposes a method to optimize policies for diverse individuals using heterogeneous data.

The paper tackles robust policy learning in MDPs using statistical methods.

Batch Thompson Sampling reduces exploration-exploitation trade-off in online decision making.