arXiv research

GRPO optimizes LLMs with verifiable rewards, amplifying policy success.

The paper proposes a new method for learning reward models from ordinal feedback, improving upon binary feedback.

Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of…

The paper enhances preference learning by incorporating response time data.

Robot learns to juggle two balls from 56 minutes of experience.

New method separates model and non-model risks for more practical asset pricing.

Thompson Sampling bounds for contextual bandits with sub-Gaussian rewards.

Sparse reward is one of the biggest challenges in reinforcement learning (RL). In this paper, we propose a novel method called Generative Exploration and Exploitation (GENE) to overcome sparse reward. GENE automatically generates start states to encourage the agent to explore the environment and to exploit received rew…

Extends linear MDP to handle nonlinear rewards.

We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often …

New method optimizes policies without assuming known link functions between preferences and rewards.

The multi-armed bandit (MAB) problem is a classic example of the exploration-exploitation dilemma. It is concerned with maximising the total rewards for a gambler by sequentially pulling an arm from a multi-armed slot machine where each arm is associated with a reward distribution. In static MABs, the reward distributi…

Sparse rewards with different magnitudes speed up learning in model-based reinforcement learning.

This work explores adaptations of successful multi-armed bandits policies to the online contextual bandits scenario with binary rewards using binary classification algorithms such as logistic regression as black-box oracles. Some of these adaptations are achieved through bootstrapping or approximate bootstrapping, whil…

Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle…

Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect)…

Neural algorithms optimize arm selection with human preference feedback for complex reward functions.

In light of the growing interest in agent-based market models, we bring together several earlier works in which we considered the topic of self-consistent market modelling. Building upon the binary game structure of Challet and Zhang, we discuss generalizations of the strategy reward scheme such that the agents seek to…

This work improves transferability of rewards inferred from expert demonstrations.

Two different formulas for macro F1 lead to significant differences in classification evaluation.

Study quantile reward identification with 1-bit feedback constraints.

Adaptive policies solve a linear program to maximize rewards while minimizing costs in sales with discounts.

New hashing method improves document retrieval precision.

Study preference-based reinforcement learning in episodic kernel MDPs.

Binary determination of the presence of objects is one of the problems where humans perform extraordinarily better than computer vision systems, in terms of both speed and preciseness. One of the possible reasons is that humans can skip most of the clutter and attend only on salient regions. Recurrent attention models …

New algorithm reduces regret for logistic bandits without $κ$ dependency.

In many settings (e.g., robotics) demonstrations provide a natural way to specify tasks; however, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the tasks, such as rewards or policies, can be safely composed and/or do not explicitly capture history dependen…

We address the problem of regret minimization in logistic contextual bandits, where a learner decides among sequential actions or arms given their respective contexts to maximize binary rewards. Using a fast inference procedure with Polya-Gamma distributed augmentation variables, we propose an improved version of Thomp…

This paper addresses the problem of learning a task from demonstration. We adopt the framework of inverse reinforcement learning, where tasks are represented in the form of a reward function. Our contribution is a novel active learning algorithm that enables the learning agent to query the expert for more informative d…

The paper analyzes RLVR's training dynamics, proving convergence depends on aligning update direction with Gradient Gap.

Optimizes reinforcement learning by prioritizing sets of samples over individual ones.

Improved sample complexity for contextual combinatorial semi-bandits with sparse rewards.

A simple DQN-based multi-agent RL system for binary actions.

SPReD uses uncertainty to decide imitation from demonstrations.

A new thompson sampling method controls for time-varying effects.

We propose combinatorial cascading bandits, a class of partial monitoring problems where at each step a learning agent chooses a tuple of ground items subject to constraints and receives a reward if and only if the weights of all chosen items are one. The weights of the items are binary, stochastic, and drawn independe…

New algorithms for GLMs adapt to non-stationary contexts.

New model tackles causal bandits with dependent variables.

The Thresholding Bandit Problem (TBP) aims to find the set of arms with mean rewards greater than a given threshold. We consider a new setting of TBP, where in addition to pulling arms, one can also \emph{duel} two arms and get the arm with a greater mean. In our motivating application from crowdsourcing, dueling two a…

Paper proposes a new RL approach combining IL and RL methods to improve decision-making.

A new algorithm for conversational recommendation systems using dueling bandits in GLMs.

We propose a new approach to inverse reinforcement learning (IRL) based on the deep Gaussian process (deep GP) model, which is capable of learning complicated reward structures with few demonstrations. Our model stacks multiple latent GP layers to learn abstract representations of the state feature space, which is link…

In this paper, we introduce an actor-critic algorithm called Deep Value Model Predictive Control (DMPC), which combines model-based trajectory optimization with value function estimation. The DMPC actor is a Model Predictive Control (MPC) optimizer with an objective function defined in terms of a value function estimat…

A scalable gradient-based framework for sparse portfolio selection.

Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.

Study on the limits of bandit learning, showing hardness and limitations.

The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it als…

Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.