arXiv research

New algorithms improve best-arm identification with varying rewards.

The paper develops a robust algorithm for contextual bandits with heavy-tailed rewards.

Reward models need more than just accuracy for effective RLHF.

A new method resolves non-identifiability in reward modeling using anchor labels.

There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator ha…

Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be a direct result of goal misspecification, randomness in the reward signal, or correlation of the rew…

Contextual bandits study how reward variance affects regret bounds.

Recent reinforcement learning algorithms, though achieving impressive results in various fields, suffer from brittle training effects such as regression in results and high sensitivity to initialization and parameters. We claim that some of the brittleness stems from variance differences, i.e. when different environmen…

New method for evaluating sequential recommendations with lower variance.

Two algorithms tackle heavy-tailed rewards in reinforcement learning with linear function approximation.

Improved GP bandit algorithms for noiseless, varying noise, and RKHS norms.

Paper tackles unknown variances in best-arm identification.

COCOA improves credit assignment in reinforcement learning by measuring contributions to rewards.

Online learning has traditionally focused on the expected rewards. In this paper, a risk-averse online learning problem under the performance measure of the mean-variance of the rewards is studied. Both the bandit and full information settings are considered. The performance of several existing policies is analyzed, an…

Study aims to optimize financial investments by balancing risk and reward efficiently.

A new game-theoretic approach balances downside risk with expected reward.

New algorithm resists corruption in linear contextual bandits.

This paper extends the MAB problem to consider risk-reward tradeoffs.

New strategy optimally identifies best arm in unknown variance Gaussian bandits.

Optimizes ranking from click feedback in a bandit setting.

Algorithm identifies best arm in bandit game with variance consideration.

Variance plays a crucial role in risk-sensitive reinforcement learning, and most risk measures can be analyzed via variance. In this paper, we consider two law-invariant risks as examples: mean-variance risk and exponential utility risk. With the aid of the state-augmentation transformation (SAT), we show that, the two…

In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of retu…

GRPO optimizes LLMs with verifiable rewards, amplifying policy success.

We study the out-of-sample properties of robust empirical optimization problems with smooth $φ$-divergence penalties and smooth concave objective functions, and develop a theory for data-driven calibration of the non-negative "robustness parameter" $δ$ that controls the size of the deviations from the nominal model. Bu…

New strategies for identifying the best arm in bandits with decreasing variances.

A new estimator reduces variance in slate bandit OPE.

New algorithms reduce contextual bandits' regret without knowing reward noise variances.

This work tackles risk-sensitive deep RL by optimizing policies with variance constraints.

Policy gradient methods are very attractive in reinforcement learning due to their model-free nature and convergence guarantees. These methods, however, suffer from high variance in gradient estimation, resulting in poor sample efficiency. To mitigate this issue, a number of variance-reduction approaches have been prop…

New estimator improves off-policy evaluation for large action spaces.

Exogenous state variables and rewards can slow down reinforcement learning by injecting uncontrolled variation into the reward signal. We formalize exogenous state variables and rewards and identify conditions under which an MDP with exogenous state can be decomposed into an exogenous Markov Reward Process involving on…

Neural-$σ^2$-LinearUCB improves regret in neural contextual bandits.

A new algorithm for offline RL with trajectory-wise reward reduces bias and variance errors.

Improves inference-time alignment for diffusion models without updating weights.

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such c…

SEMI uses multisensory incongruity to self-supervise exploration in reinforcement learning.

Thompson sampling used for linear bandits with normal-gamma priors.

This study examines deep hedging for S&P 500 options, revealing systematic delta corrections and fragility.

We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance p…

The paper tackles reinforcement learning with exogenous variables and rewards.

This paper considers the multi-armed thresholding bandit problem -- identifying all arms whose expected rewards are above a predefined threshold via as few pulls (or rounds) as possible -- proposed by Locatelli et al. [2016] recently. Although the proposed algorithm in Locatelli et al. [2016] achieves the optimal round…

This work classifies SOC loss functions based on their gradient properties.

PROBE optimizes best-arm identification with cheap proxies, improving sample complexity.

We study decision making in environments where the reward is only partially observed, but can be modeled as a function of an action and an observed context. This setting, known as contextual bandits, encompasses a wide variety of applications including health-care policy and Internet advertising. A central task is eval…

In this paper we extend temporal difference policy evaluation algorithms to performance criteria that include the variance of the cumulative reward. Such criteria are useful for risk management, and are important in domains such as finance and process control. We propose both TD(0) and LSTD(lambda) variants with linear…

PFN-TS uses Thompson sampling with PFNs to improve contextual bandit performance.

We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …