Method tackles uncertainty in reward models for LLMs from heterogeneous human feedback.
problem Uncertainty in reward models for LLMs from heterogeneous human feedback.
method Heterogeneous preference framework and alternating gradient descent algorithm.
result Established theoretical guarantees for estimator convergence and asymptotic distribution.
A new reward shaping method balances learning efficiency and effectiveness for robot manipulation.
problem Efficient and effective learning in robot manipulations with system uncertainty.
method Dense2Sparse reward shaping method combining dense and sparse rewards.
result Dense2Sparse method achieves higher expected reward and better system uncertainty tolerance.
Bayesian nonparametric models improve multi-armed bandit performance with uncertain rewards.
problem Reward model uncertainty in multi-armed bandits.
method Bayesian nonparametric Gaussian mixture models for flexible reward density estimation.
result Achieves successful regret performance with asymptotic regret bound.
PROWL uses robust reward estimates to improve ITR selection.
problem Reward uncertainty in ITR estimation leads to inflated performance.
method PAC-Bayesian framework with reward uncertainty certificates.
result PROWL achieves better robust treatment regime estimation.
Directed exploration improves reinforcement learning efficiency and robustness.
problem Achieving good sample efficiency in reinforcement learning with efficient exploration.
method Directed exploration through goal-conditioned policies that are independent of uncertainty.
result Directed exploration is more efficient and robust to uncertainty than reward bonuses.
This work uses model uncertainty for efficient exploration in sparse reward environments.
problem Challenging exploration in sparse reward reinforcement learning environments.
method Implicit generative modeling approach to estimate Bayesian uncertainty of the agent's belief of the environment dynamics.
result Our implicit generative model consistently outperforms competing approaches in data efficiency for exploration.
New method reduces uncertainty in Q-values for better reinforcement learning.
problem Uncertainty in expected future rewards in reinforcement learning.
method Introduces knowledge values that combine expected rewards and uncertainty.
result Achieves Bayes regret bound of O(L √SA T).
Improved DPO framework penalizes preference uncertainty to avoid overoptimization.
problem Aligning LLMs to human preferences is challenging due to varied, context-dependent, and ambiguous preferences.
method Developed a pessimistic framework for DPO by introducing preference uncertainty penalization schemes.
result Improved overall performance and better completions on high-uncertainty responses compared to vanilla DPO.
Study designs steering rewards for MFGs with unknown dynamics and model uncertainty.
problem Designing incentives for large populations of agents in MFGs with uncertain model details.
method Developed optimistic exploration algorithms for agents with no-adaptive regret behaviors.
result Sub-linear regret guarantees for cumulative gaps between agent behaviors and desired outcomes.
Paper improves SLCB regret bound for bounded noise.
problem Stochastic linear contextual bandits with bounded noise.
method Set-membership estimation (SME) and optimism in the face of uncertainty (OFU).
result Improved regret bound of O(logT). Paper tackles robust offline RL with heavy-tailed rewards.
problem Real-world applications often encounter heavy-tailed rewards, challenging offline RL.
method Proposes ROAM and ROOM algorithms using median-of-means method for robust off-policy evaluation and OPO.
result Demonstrates superior performance on heavy-tailed reward datasets compared to existing methods.
Study improves reinforcement learning for stable long-term performance.
problem Distributionally robust average-reward reinforcement learning for stable long-term performance.
method Proposes two algorithms to achieve near-optimal sample complexity.
result Achieves a sample complexity of O(∣S∣∣A∣tmix2ε−2) for estimating optimal policy and robust average reward. Novel framework for risk-sensitive reinforcement learning using martingale decomposition.
problem Risk sensitivity in sequential decision-making with uncertain rewards.
method Martingale decomposition and chaotic variation for reward uncertainty, integrated into model-free reinforcement learning algorithms.
result Demonstrated relevance of risk-sensitive reinforcement learning in grid world and portfolio optimization problems.
Unified approach to RLHF tackles uncertainty in reward function.
problem Uncertainty in reward function learned from human feedback.
method Value-incentivized preference optimization (VPO) that regularizes the reward function with value function.
result Theoretical and practical guarantees for both online and offline RLHF settings.
Efficient exploration improves large language model performance with fewer queries.
problem Improving large language model performance with fewer human feedback queries.
method Sequentially generates queries, fits a reward model to feedback, uses double Thompson sampling with epistemic neural network uncertainty.
result Efficient exploration enables high performance with far fewer queries.
B-REX efficiently learns Atari game policies from pixel inputs using Bayesian methods.
problem Learning reward functions from visual inputs with uncertainty and safety considerations.
method Bayesian Reward Extrapolation (B-REX) using successor features and preferences.
result B-REX generates posterior samples efficiently, enabling high-confidence performance bounds.
CARDS improves decoding efficiency and alignment quality for LLMs.
problem Efficiency bottlenecks in decoding-time alignment for LLMs.
method Cascade Reward Sampling (CARDS) with segment-level rejection sampling and uncertainty-based segmentation.
result Significant improvement in decoding efficiency and alignment quality.
Book introduces multi-armed bandits for decision-making under uncertainty.
problem Decision-making under uncertainty with limited information.
method Self-contained chapters covering various types of bandits.
result Provides a comprehensive introduction to multi-armed bandits.
New method identifies uncertainty shocks in financial markets using revised VIX.
problem Traditional VIX fails to capture non-Gaussian, heavy-tailed asset returns.
method Fit a double-subordinated Normal Inverse Gaussian Levy process to S&P 500 option prices to construct a revised VIX.
result Revised VIX provides a more comprehensive measure of volatility reflecting extreme movements and heavy tails.
SPReD uses uncertainty to decide imitation from demonstrations.
problem Learning from sparse rewards with few demonstrations.
method Ensemble methods to model Q-value distributions, probabilistic and advantage-based uncertainty quantification.
result Significant gains in reinforcement learning across multiple tasks.
New algorithm for RL with horizon-free reward-free exploration for linear MDPs.
problem Reward-free reinforcement learning with long planning horizons.
method Uncertainty-weighted value-targeted regression with exploration-driven pseudo-reward and moment estimator.
result Horizon-free sample complexity of O(d2ε−2) for finding an ε-optimal policy. Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.
Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.
problem Quantifying uncertainty in off-policy estimates for safe deployment of target policies.
method Distributionally robust optimization for creating confidence bounds.
result Non-asymptotic and asymptotic guarantees for robust cumulative reward estimates.
Generative Flow Networks use submodular upper bounds to generate more data.
problem Generating data from unknown, complex reward functions efficiently.
method Introduce submodular upper bounds to estimate reward, use Optimism in the Face of Uncertainty principle to train GFNs.
result SUBo-GFN generates significantly more data than classical GFNs.
AdaDEM decouples EM into two parts to improve class overlap and uncertainty.
problem Improper EM limits its effectiveness in various machine learning tasks.
method Decouple EM into CADF and GMC, and AdaDEM normalizes CADF reward and uses MEC.
result AdaDEM outperforms classical EM and improves performance in noisy and dynamic environments.
Polynomial chaos surrogates quantify epistemic uncertainty in AI-driven scientific models.
problem Uncertainty in reward estimates hinders interpretability in sequential generative models.
method Fit polynomial chaos expansions to trained models to propagate epistemic uncertainty and quantify sensitivity.
result Interpretable decomposition of reward components driving generative decisions.
New method separates model and non-model risks for more practical asset pricing.
problem Asset pricing under model-uncertainty.
method Binary model-risks and constraints over preferences; unique model-risk pricing formula.
result Unique model-risk pricing formula with dynamically conserved constant.
Bayesian Robust Optimization for Imitation Learning (BROIL) balances risk and reward.
problem Learning robust policies for new states in imitation learning.
method Bayesian reward function inference and user-specific risk tolerance.
result BROIL outperforms risk-sensitive and risk-neutral algorithms.
This study optimizes energy storage scheduling under price uncertainty, balancing risk and reward.
problem Optimizing energy storage operation under price uncertainty and risk.
method Two-stage stochastic risk-constrained approach using conditional value-at-risk.
result Increasing risk aversion leads to substantial benefits in terms of risk reduction and expected reward.
Improved reinforcement learning for robotics with active uncertainty reduction.
problem Infeasibility of model-free reinforcement learning methods in robotics due to safety and time constraints.
method Active uncertainty reduction-based virtual environments with adaptive sampling for metric self-improvement.
result Better modeling capacity for complex system dynamics compared to established methods.
New method for efficient online exploration in RLHF reduces regret.
problem Efficiently collecting new preference data in RLHF to refine reward model and policy.
method Proposes a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences most relevant to policy improvement.
result Establishes regret bounds of order T(β+1)/(β+2) for online RLHF, with polynomial scaling in all model parameters. POETS optimizes LLMs by combining policy ensembles and KL regularization.
problem Balancing exploration and exploitation in decision-making and optimization.
method POETS uses policy ensembles and KL regularization to optimize LLMs efficiently.
result POETS achieves state-of-the-art sample efficiency across various domains.
New method boosts skill learning by encouraging optimistic exploration.
problem Intrinsic reward for exploration is inherently pessimistic.
method Derive an information gain auxiliary objective involving an ensemble of discriminators and rewarding policy disagreement.
result Improves skill learning in grid worlds and Atari games.
Bayesian REX learns Atari games from demonstrations efficiently.
problem Bayesian reward learning for complex control problems is computationally intractable.
method Bayesian Reward Extrapolation (Bayesian REX) pre-trains a low-dimensional feature encoding and uses preferences to perform fast Bayesian inference.
result Bayesian REX learns Atari games from demonstrations in 5 minutes, competitive with state-of-the-art methods.
EBRM improves robustness and generalization of language model rewards.
problem Challenges in capturing complex human preferences and generalizing to unseen data in reward models.
method Energy-Based Reward Model (EBRM) that models reward distribution explicitly, using conflict-aware data filtering, label-noise-aware contrastive training, and hybrid initialization.
result Significant improvements in robustness and generalization, up to 5.97% improvement in safety-critical alignment tasks.
Method predicts future rewards from past actions in a linear Gaussian system.
problem Maximizing cumulative reward in a stochastic multi-armed bandit with linear Gaussian dynamics.
method Proposes a method using a modified Kalman filter to predict future rewards based on past rewards.
result Reward from any action can be used to predict another action's future reward.
New Thompson sampling uses local uncertainty for better decision making.
problem Sequential decision making with exploration-exploitation dilemma.
method Proposes a new probabilistic modeling framework using local latent variable uncertainty for Thompson sampling, with variational inference and semi-implicit structure.
result Thompson sampling guided by local uncertainty achieves state-of-the-art performance with low computational complexity.
New method quantifies uncertainty in reinforcement learning models.
problem Quantifying uncertainty over expected cumulative rewards in reinforcement learning.
method Proposes a new uncertainty Bellman equation to more accurately estimate value function variance.
result Our method converges to the true posterior variance over values and improves sample-efficiency.
New framework for efficient query-based imitation learning.
problem Aligning agent policy with human expert behavior without prior knowledge.
method Adversarial reward query with successor representation.
result Significantly outperforms uncertainty-based methods in query efficiency.
New algorithms optimize decision-making under uncertainty with graph information.
problem Optimizing decisions in large, uncertain environments with graph-based similarities.
method Graph-based UcB and ζ-UcB algorithms for maximizing and satisficing.
result Proves algorithms are near-optimal and benefits from graph side information.
MaxEnt RL optimizes decision-making in uncertain environments.
problem MaxEnt RL optimizes decision-making in uncertain environments.
method Formal analysis of MaxEnt RL in POMDPs and adversarial reward games.
result MaxEnt RL optimally solves certain classes of control problems with uncertain reward functions.
A new RL method boosts exploration efficiency in Atari games and driving simulators.
problem Efficient exploration in reinforcement learning environments.
method A two-component approach: decaying schedule and upper quantile bonus.
result Significantly outperforms QR-DQN in Atari games and near-optimal safety in CARLA.
This work compares human feedback methods for reward learning in bandits.
problem Understanding how human feedback affects the performance of reward learning methods.
method Theoretical comparison of human feedback approaches in offline contextual bandits.
result Human bias and uncertainty in feedback modeling impact the theoretical guarantees of reward learning methods.
Trains neural nets for gamma hedging with model uncertainty.
problem Gamma hedging with model mismatch.
method Trains neural networks using loss functions that reward model uncertainty.
result Networks can learn optimal gamma hedging even with model mismatch.
Efficient algorithms exploit structure of uncertainty for combinatorial semi-bandits.
problem Optimizing algorithms for stochastic combinatorial semi-bandits with structural properties.
method Reduction to submodular maximization, adapted approximation routines for matroid constraints.
result Improved efficient gap-free regret bound by a factor of sqrt(m)/log m.
Algorithm identifies optimal stable matching in uncertain two-sided markets.
problem Sequential learning in two-sided markets with unknown preferences.
method Pure exploration approach with elimination-based algorithms exploiting partial preference information.
result Identification of pervasive stable matching for optimal stable matching identification.
In a multi-armed bandit (MAB) problem a gambler needs to choose at each round of play one of K arms, each characterized by an unknown reward distribution. Reward realizations are only observed when an arm is selected, and the gambler's objective is to maximize his cumulative expected earnings over some given horizon of…
IDS optimizes regret in stochastic partial monitoring with linear rewards.
problem Optimizing decision-making in uncertain environments with linear rewards.
method Information Directed Sampling (IDS) for stochastic partial monitoring.
result Achieves optimal regret rates in all observable game regimes.