arXiv research

New acquisition function for extreme rewards in bandits.

A new method optimizes in nonstationary environments with many arms efficiently.

Algorithms for hyperparameter optimization abound, all of which work well under different and often unverifiable assumptions. Motivated by the general challenge of sequentially choosing which algorithm to use, we study the more specific task of choosing among distributions to use for random hyperparameter optimization.…

Reward models need more than just accuracy for effective RLHF.

Study efficient resource allocation for detecting extreme values.

Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.

A new EM framework for goal-conditioned RL improves performance on sparse reward tasks.

A new algorithm for selecting top-k arms in extreme contextual bandits with improved efficiency.

The paper has 2 main goals: 1. We propose a variant of the CAPM based on coherent risk. 2. In addition to the real-world measure and the risk-neutral measure, we propose the third one: the extreme measure. The introduction of this measure provides a powerful tool for investigating the relation between the first two mea…

New findings on universal learning in contextual bandits with adversarial rewards.

The paper tackles catastrophic risk in reinforcement learning using extreme value theory.

Learning about many things can provide numerous benefits to a reinforcement learning system. For example, learning many auxiliary value functions, in addition to optimizing the environmental reward, appears to improve both exploration and representation learning. The question we tackle in this paper is how to sculpt th…

Vanishing gradients hinder reinforcement finetuning of language models.

Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.

New bandit algorithm detects and adapts to seasonal changes in rewards.

We propose the first fully-adaptive algorithm for pure exploration in linear bandits---the task to find the arm with the largest expected reward, which depends on an unknown parameter linearly. While existing methods partially or entirely fix sequences of arm selections before observing rewards, our method adaptively c…

We investigate the optimality of perturbation based algorithms in the stochastic and adversarial multi-armed bandit problems. For the stochastic case, we provide a unified regret analysis for both sub-Weibull and bounded perturbations when rewards are sub-Gaussian. Our bounds are instance optimal for sub-Weibull pertur…

Bayesian Robust Optimization for Imitation Learning (BROIL) balances risk and reward.

This thesis tackles learning reward functions from human comparative feedback.

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agen…

Exploration is an extremely challenging problem in reinforcement learning, especially in high dimensional state and action spaces and when only sparse rewards are available. Effective representations can indicate which components of the state are task relevant and thus reduce the dimensionality of the space to explore.…

Framework for optimizing search engine rankings using observational data.

Identifies arms with rewards significantly different from the majority.

Multi-step temporal difference (TD) learning is an important approach in reinforcement learning, as it unifies one-step TD learning with Monte Carlo methods in a way where intermediate algorithms can outperform either extreme. They address a bias-variance trade off between reliance on current estimates, which could be …

We explore the use of Evolution Strategies (ES), a class of black box optimization algorithms, as an alternative to popular MDP-based RL techniques such as Q-learning and Policy Gradients. Experiments on MuJoCo and Atari show that ES is a viable solution strategy that scales extremely well with the number of CPUs avail…

New method reduces bias in learning from large action spaces using selective importance sampling.

New method identifies uncertainty shocks in financial markets using revised VIX.

We study a nonparametric contextual bandit problem where the expected reward functions belong to a Hölder class with smoothness parameter $β$. We show how this interpolates between two extremes that were previously studied in isolation: non-differentiable bandits ($β\leq1$), where rate-optimal regret is achieved by run…

In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in a…

It is well known that quantile regression model minimizes the portfolio extreme risk, whenever the attention is placed on the estimation of the response variable left quantiles. We show that, by considering the entire conditional distribution of the dependent variable, it is possible to optimize different risk and perf…

New meta-learning approach for bandit policies that achieve high average reward.

Optimizes forecast distributions for financial risk management.

An agent that has well understood the environment should be able to apply its skills for any given goals, leading to the fundamental problem of learning the Universal Value Function Approximator (UVFA). A UVFA learns to predict the cumulative rewards between all state-goal pairs. However, empirically, the value functio…

New RLHF approach mitigates bias in aligning LLMs with human preferences.

Deep reinforcement learning techniques have demonstrated superior performance in a wide variety of environments. As improvements in training algorithms continue at a brisk pace, theoretical or empirical studies on understanding what these networks seem to learn, are far behind. In this paper we propose an interpretable…

Entropy regularization improves policy optimization in reinforcement learning.

Managers of US National Forests must decide what policy to apply for dealing with lightning-caused wildfires. Conflicts among stakeholders (e.g., timber companies, home owners, and wildlife biologists) have often led to spirited political debates and even violent eco-terrorism. One way to transform these conflicts into…

The security of Deep Reinforcement Learning (Deep RL) algorithms deployed in real life applications are of a primary concern. In particular, the robustness of RL agents in cyber-physical systems against adversarial attacks are especially vital since the cost of a malevolent intrusions can be extremely high. Studies hav…

Modulating masks improve lifelong reinforcement learning.

Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.

We consider Bandits with Knapsacks (henceforth, BwK), a general model for multi-armed bandits under supply/budget constraints. In particular, a bandit algorithm needs to solve a well-known knapsack problem: find an optimal packing of items into a limited-size knapsack. The BwK problem is a common generalization of nume…

Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.

This work analyzes the value of future reward information in RL.

Self-supervised reward prediction improves RL in sparse reward settings.

The study categorizes reward errors in reinforcement learning, finding some can be beneficial.

Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.

Proposes a method to boost deep reinforcement learning with sparse rewards.

Action guidance helps agents learn true objectives in games with sparse rewards.