arXiv research

New lower bounds for combinatorial multi-armed bandits for general reward functions.

We consider the combinatorial multi-armed bandit (CMAB) problem, where the reward function is nonlinear. In this setting, the agent chooses a batch of arms on each round and receives feedback from each arm of the batch. The reward that the agent aims to maximize is a function of the selected arms and their expectations…

A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a …

Imitation Learning describes the problem of recovering an expert policy from demonstrations. While inverse reinforcement learning approaches are known to be very sample-efficient in terms of expert demonstrations, they usually require problem-dependent reward functions or a (task-)specific reward-function regularizatio…

In this paper we consider the problem of online stochastic optimization of a locally smooth function under bandit feedback. We introduce the high-confidence tree (HCT) algorithm, a novel any-time $\mathcal{X}$-armed bandit algorithm, and derive regret bounds matching the performance of existing state-of-the-art in term…

Consider a nonparametric contextual multi-arm bandit problem where each arm $a \in [K]$ is associated to a nonparametric reward function $f_a: [0,1] \to \mathbb{R}$ mapping from contexts to the expected reward. Suppose that there is a large set of arms, yet there is a simple but unknown structure amongst the arm reward…

We consider a discounted reward control problem in continuous time stochastic environment where the discount rate might be an unbounded function of the control process. We provide a set of general assumptions to ensure that there exists a smooth classical solution to the corresponding HJB equation. Moreover, some verif…

New algorithm optimizes smooth functions with Hölder exponent > 1.

IDS improves RLHF by smoothing reward data, enhancing model performance.

In structured output prediction tasks, labeling ground-truth training output is often expensive. However, for many tasks, even when the true output is unknown, we can evaluate predictions using a scalar reward function, which may be easily assembled from human knowledge or non-differentiable pipelines. But searching th…

Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.

Improved inference-time alignment using Best-of-N and smoothing.

Optimal strategy proposed for maximizing cumulative reward in continuum-armed bandits.

This paper proposes a new algorithm for learning guidance rewards in RL.

BERT embeddings improve sequence quality metrics.

We study the out-of-sample properties of robust empirical optimization problems with smooth $φ$-divergence penalties and smooth concave objective functions, and develop a theory for data-driven calibration of the non-negative "robustness parameter" $δ$ that controls the size of the deviations from the nominal model. Bu…

Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient…

A new ES method improves reinforcement learning speed and accuracy.

In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…

We consider a multi-armed bandit problem in a setting where each arm produces a noisy reward realization which depends on an observable random covariate. As opposed to the traditional static multi-armed bandit problem, this setting allows for dynamically changing rewards that better describe applications where side inf…

New method extends low-rank MDPs to continuous action spaces.

Adaptive smooth non-stationary bandits achieve optimal regret rates without knowing parameters.

New reward function improves GAIL performance in task-based environments.

This paper improves sample efficiency for off-policy evaluation with preference data.

Many reinforcement-learning researchers treat the reward function as a part of the environment, meaning that the agent can only know the reward of a state if it encounters that state in a trial run. However, we argue that this is an unnecessary limitation and instead, the reward function should be provided to the learn…

This work characterizes reward function partial identifiability and its impact on policy optimization.

Unified framework for multi-user bandits using Laplacian kernels.

Self-supervised reward prediction improves RL in sparse reward settings.

New concept: reward hacking, where optimizing a flawed reward function can hurt performance.

We present a novel method for learning a set of disentangled reward functions that sum to the original environment reward and are constrained to be independently obtainable. We define independent obtainability in terms of value functions with respect to obtaining one learned reward while pursuing another learned reward…

Enhances reward specification in RL with a novel language-based approach.

EPIC quantifies reward differences without policy optimization.

New algorithm for reward-free RL with linear function approximation, reducing sample complexity.

Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus…

We consider the sequential Bayesian optimization problem with bandit feedback, adopting a formulation that allows for the reward function to vary with time. We model the reward function using a Gaussian process whose evolution obeys a simple Markov model. We introduce two natural extensions of the classical Gaussian pr…

PQR estimates reward functions from actions and states without assuming state-only rewards.

In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem, or close variants thereof, have been proposed in the literature. In this…

Paper proposes operator deep Q-learning for quick reward adaptation.

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create age…

In this paper, we introduce a novel combined reward cum penalty loss function to handle the regression problem. The proposed combined reward cum penalty loss function penalizes the data points which lie outside the $ε$-tube of the regressor and also assigns reward for the data points which lie inside of the $ε$-tube of…

Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.

Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.

New algorithm finds optimal policies without knowing reward functions.

Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.

AIRL learns robust, generalizable reward functions from demonstrations.

New RL method explores environments without rewards, achieving efficient policy generation.

This paper introduces a new reward shaping method for average-reward reinforcement learning.

Learning reward functions can lead to poor policy performance despite low error.