arXiv research

Extends reinforcement learning alignment to scalar rewards, improving math reasoning.

DPA aligns LLMs with multi-objective rewards for diverse user preferences.

DOPL learns from preference feedback to solve RMAB problems.

Enhances reward specification in RL with a novel language-based approach.

Paper establishes DRL for high-dimensional rewards.

Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor ma…

New algorithms for multivariate RL improve decision-making in complex systems.

New algorithm for fair ranking in contextual bandits with concave rewards.

This work overcomes bias in concave multi-objective reinforcement learning.

In structured output prediction tasks, labeling ground-truth training output is often expensive. However, for many tasks, even when the true output is unknown, we can evaluate predictions using a scalar reward function, which may be easily assembled from human knowledge or non-differentiable pipelines. But searching th…

OP-GFNs sample candidates in order-preserving proportion to a learned reward function.

New method optimizes policies without assuming known link functions between preferences and rewards.

The stochastic multi-armed bandit (MAB) problem is a common model for sequential decision problems. In the standard setup, a decision maker has to choose at every instant between several competing arms, each of them provides a scalar random variable, referred to as a "reward." Nearly all research on this topic consider…

We study an online decision making problem where on each round a learner chooses a list of items based on some side information, receives a scalar feedback value for each individual item, and a reward that is linearly related to this feedback. These problems, known as contextual semibandits, arise in crowdsourcing, rec…

Duel-Evolve uses LLM self-preferences for test-time optimization of discrete outputs.

Paper presents a modular RL framework for Forex trading, addressing limitations of prior studies.

A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a …

Develops neural network framework for risk-reward optimization problems.

Paper characterizes optimal language model alignment methods.

Model user preferences for conversational LLMs using weak rewards.

Trajectory-level supervision allows efficient offline reinforcement learning.

New algorithm catches moving subspaces in bandit problems.

Signed compression progress on a sealed audit is goodhart-resistant.

This paper analyzes MORL and proposes efficient algorithms to learn Pareto optimal policies.

The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically…

Q-MMR evaluates policies using reweighted rewards and moment matching.

New method reduces total cost constraints in CBwK to sqrt(T) with fairness application.

Policy Gradient (PG) algorithms are among the best candidates for the much-anticipated applications of reinforcement learning to real-world control tasks, such as robotics. However, the trial-and-error nature of these methods poses safety issues whenever the learning process itself must be performed on a physical syste…

Multi-objective Neural Architecture Search (NAS) aims to discover novel architectures in the presence of multiple conflicting objectives. Despite recent progress, the problem of approximating the full Pareto front accurately and efficiently remains challenging. In this work, we explore the novel reinforcement learning …

Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.

Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.

This work analyzes the value of future reward information in RL.

Self-supervised reward prediction improves RL in sparse reward settings.

The study categorizes reward errors in reinforcement learning, finding some can be beneficial.

Reward collapse occurs when ranking-based reward models yield uniform rewards for different prompts.

Reward models need more than just accuracy for effective RLHF.

Action guidance helps agents learn true objectives in games with sparse rewards.

Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.

Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond …

Reward tweaking optimizes behavior for long-term goals by adjusting the reward function.

We propose a generic, Bayesian, information geometric approach to the exploration--exploitation trade-off in multi-armed bandit problems. Our approach, BelMan, uniformly supports pure exploration, exploration--exploitation, and two-phase bandit problems. The knowledge on bandit arms and their reward distributions is su…

Many reinforcement-learning researchers treat the reward function as a part of the environment, meaning that the agent can only know the reward of a state if it encounters that state in a trial run. However, we argue that this is an unnecessary limitation and instead, the reward function should be provided to the learn…

This work characterizes reward function partial identifiability and its impact on policy optimization.

This paper introduces a new reward shaping method for average-reward reinforcement learning.

While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an ongoing challenge. In this work, we propose an effective reward shaping method through predictive coding to tackle sparse reward problems. By learning predictive repr…

Paper generalizes reward distribution in multi-armed bandits with temporally-partitioned rewards.

We present a novel method for learning a set of disentangled reward functions that sum to the original environment reward and are constrained to be independently obtainable. We define independent obtainability in terms of value functions with respect to obtaining one learned reward while pursuing another learned reward…

New reward function improves GAIL performance in task-based environments.