New concept: reward hacking, where optimizing a flawed reward function can hurt performance.
problem Optimizing imperfect reward functions leads to poor performance.
method Formal definition and analysis of reward hacking, examining conditions for unhackability.
result Reward functions are usually hackable, making it hard to align AI with human values.
New conditions show proxy data can improve policy learning in sparse expert data contexts.
problem Improving sample complexity of policy learning with sparse expert data.
method Sufficient conditions for proxy data to improve sample complexity, adaptable to LLMs.
result Provable improvement in sample complexity for policy learning with proxy data.
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
PROBE optimizes best-arm identification with cheap proxies, improving sample complexity.
problem Fixed-confidence best-arm identification with costly rewards and correlated cheap proxies.
method PROBE uses control-variate adjustment and phase elimination to learn residual variance online.
result PROBE achieves oracle sample complexity up to a constant factor and additive calibration cost.
Study scaling laws of reward model overoptimization in reinforcement learning.
problem Reward model overoptimization hinders true performance in reinforcement learning.
method Synthetic setup with fixed gold reward model; optimization using RL or best-of-n sampling; analysis of scaling laws. result Scaling laws of reward model overoptimization differ based on optimization method and scale smoothly with model parameters.
Improved inference-time alignment using Best-of-N and smoothing.
problem Reward overoptimization in Best-of-N (BoN) due to poor proxy reward models.
method Introduced Soft Best-of-N (SBoN) and analyzed its performance through KL divergence and regret analysis.
result Smoothing helps SBoN mitigate reward overoptimization, especially when proxy reward quality is low.
Research suggests using deep learning for better recommendation systems.
problem Recommender systems rely on proxies for A/B testing, leading to random success.
method Advocates for using deep learning to improve recommendation performance.
result Deep learning can potentially optimize reward in recommendation systems.
A new algorithm for offline RL with trajectory-wise reward reduces bias and variance errors.
problem Offline RL with trajectory-wise reward incurs large bias and variance errors.
method PARTED algorithm that decomposes trajectory return into proxy rewards and performs pessimistic value iteration.
result PARTED achieves provably efficient suboptimality bounds in general MDPs with trajectory-wise reward.
The paper analyzes the reward improvement of aligned policies in large language models.
problem Optimizing policies in large language models while staying close to a reference policy.
method Information-theoretic analysis and reduction to exponential order statistics.
result Information-theoretic upper bounds on reward improvement are derived.
New algorithm optimizes for long-term user satisfaction in delayed reward settings.
problem Optimizing for long-term user satisfaction in delayed reward settings.
method Developed a predictive model of delayed rewards and a bandit algorithm that combines rewards and surrogate outcomes.
result Our algorithm significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards.
Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
Improved DPO framework penalizes preference uncertainty to avoid overoptimization.
problem Aligning LLMs to human preferences is challenging due to varied, context-dependent, and ambiguous preferences.
method Developed a pessimistic framework for DPO by introducing preference uncertainty penalization schemes.
result Improved overall performance and better completions on high-uncertainty responses compared to vanilla DPO.
Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus…
Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
New algorithm optimizes long-term user satisfaction in recommendation systems.
problem Optimizing long-term user satisfaction in recommendation systems with delayed rewards.
method Developed a predictive model of delayed rewards and a bandit algorithm that balances exploration and exploitation.
result Our approach results in substantially better performance compared to short-term or delayed optimization.
The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. …
Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Furthermore, relying solely on the a…
We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and …
Proper balance between exploitation and exploration is what makes good decisions, which achieve high rewards like payoff or evolutionary fitness. The Infomax principle postulates that maximization of information directs the function of diverse systems, from living systems to artificial neural networks. While specific a…
PROWL uses robust reward estimates to improve ITR selection.
problem Reward uncertainty in ITR estimation leads to inflated performance.
method PAC-Bayesian framework with reward uncertainty certificates.
result PROWL achieves better robust treatment regime estimation.
Study tackles OPE in confounded settings, estimating policy value from proxies.
problem Difficulty in OPE due to unobserved confounders in infinite-horizon RL.
method Two-stage approach: estimating stationary distribution ratios and combining optimal balancing.
result Policy value can be identified from off-policy data with proxies and latent variable model.
The study analyzes how neural reward models learn features for policy optimization in a Gaussian single-index model.
problem Reward modeling in policy optimization and its impact on downstream value.
method Two-stage neural reward model: first learns hidden direction, then fits readout layer.
result For any feature-learning temperature above a dimension-free threshold, a constant fraction of neurons recover the hidden direction.
PWIL learns agent behavior from expert using Wasserstein distance.
problem Learning agent behavior from expert demonstrations efficiently.
method Primal Wasserstein Imitation Learning (PWIL) with offline reward function.
result PWIL achieves expert behavior on MuJoCo tasks with minimal interactions.
Quantum model outperforms classical in training but underperforms in real-world metrics.
problem Mismatch between proxy reward signals and true investment objectives in financial domains.
method Hybrid quantum-classical reinforcement learning framework with automated feature engineering.
result Quantum models achieve higher training rewards but underperform in real-world metrics.
A new framework improves VaR recalibration by balancing reliance on imperfect volatility proxies.
problem How to balance reliance on imperfect volatility proxies in one-sided VaR recalibration.
method Proxy-reliance control framework that interpolates between constant-shift and proxy-scaled corrections.
result Lower or intermediate proxy reliance can outperform fully proxy-scaled recalibration in stressed left-tail VaR control.
Study accelerates NAS research with a large dataset of ZC proxies.
problem Speeding up neural architecture search with ZC proxies.
method Created NAS-Bench-Suite, evaluated 13 ZC proxies across 28 tasks, and provided a unified codebase.
result ZC proxies capture substantial complementary information and can improve NAS algorithm performance.
Unified framework for analyzing neural networks trained by gradient descent.
problem Lack of generalizable guarantees for neural networks trained by gradient descent.
method Proxy convexity and proxy Polyak-Lojasiewicz inequalities.
result Unified guarantees for neural networks trained by gradient descent.
Mitigates overoptimization in RLHF by reformulating SFT loss as a preference optimization loss.
problem Overoptimization in RLHF where reward model misguides generative model.
method Proposes a theoretical algorithm that minimizes maximum likelihood estimation and reward penalty, reformulates as simple objective combining preference optimization and supervised learning losses.
result Improved performance of RPO compared to DPO baselines in aligning LLMs.
New method optimizes diffusion models without fine-tuning, integrating soft value functions.
problem Optimizing natural design spaces of images, molecules, DNA, RNA, and protein sequences.
method Iterative sampling method integrating soft value functions into diffusion model inference.
result Directly utilizes non-differentiable features/reward feedback, applies to discrete diffusion models.
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
problem Risk-sensitive reinforcement learning with downside risk constraints.
method Introduce a new Bellman equation to estimate the lower partial moment of returns, use natural policy gradients, and extend Reward Constrained Policy Optimization.
result Sample-efficient estimation of partial moments and effective risk-sensitive control.
Study semi-supervised learning with noisy proxy covariates, deriving bounds and showing gains.
problem Learning from noisy proxy covariates with scarce labels.
method Two-stage estimator learning kernel eigenfeatures from all proxy covariates and fitting a ridge predictor on labeled data.
result Finite sample bounds show fast labeled sample rates and consistent gains over supervised and semi-supervised baselines.
DFPV improves PCL for confounded bandit policy evaluation.
problem Estimating causal effects in confounded settings with high-dimensional data.
method Deep feature proxy variable method (DFPV) for high-dimensional, nonlinear relationships.
result DFPV outperforms state-of-the-art methods on synthetic benchmarks and confounded bandit problems.
A framework uses proxies to prioritize treatment without estimating causal effects.
problem Prioritizing treatment when causal effects are hard to estimate.
method Decision-focused framework identifying conditions for proxy usefulness.
result Proxies can recover correct effect ordering under specific conditions.
New method recovers latent confounders from high-dimensional proxy variables.
problem Detecting latent confounders from high-dimensional proxy variables.
method Proxy Confounder Factorization (PCF) framework using ICA-PCF and GD-PCF.
result ICA-PCF recovers confounders with high correlation and low error in synthetic and real-world data.
Single proxy variable helps estimate causal effects from confounders.
problem Estimating causal effects from treatment to outcome when unobserved confounders are present.
method Assumes a single, potentially multi-dimensional proxy variable of the unobserved confounder and a known mechanism generating the proxy from the confounder. Proves causal effects are identifiable under completeness assumption.
result Causal effects are identifiable under SPICE assumption.
Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated…
A scalable method for deep metric learning using chance constraints.
problem Improving deep metric learning by addressing feasibility issues.
method Relating DML to chance constraints, reformulating as a feasibility problem, and iteratively training proxies.
result The method effectively improves deep metric learning performance across multiple benchmarks.
Develops methods to improve demand counterfactuals from imperfect proxies.
problem Imperfect proxies in demand models lead to biased counterfactuals and invalid inference.
method Practical toolkit for market-level and individual data, requiring minimal computation.
result Improves substitution prediction and counterfactual performance.
New method handles many noisy proxy controls for causal inference.
problem Causal inference with many noisy proxy controls and unknown confounders.
method Linear models with rank-restricted and sparse nuisance parameters, penalization methods.
result Estimators achieve better performance in high dimensions, especially with many proxies.
Predicting delayed outcomes is an important problem in recommender systems (e.g., if customers will finish reading an ebook). We formalize the problem as an adversarial, delayed online learning problem and consider how a proxy for the delayed outcome (e.g., if customers read a third of the book in 24 hours) can help mi…
A machine learning model may exhibit discrimination when used to make decisions involving people. One potential cause for such outcomes is that the model uses a statistical proxy for a protected demographic attribute. In this paper we formulate a definition of proxy use for the setting of linear regression and present …
A new probabilistic approach improves deep metric learning by considering image uncertainties and class-specific variances.
problem Proxy-based deep metric learning struggles with image uncertainties and class-specific structures.
method Introduces non-isotropic probabilistic proxy-based deep metric learning using directional von Mises-Fisher distributions.
result Improves generalization performance and competitive on standard benchmarks.
This paper improves auto-augment efficiency by sharing augmentation weights.
problem Efficient evaluation of augmentation policies for model training.
method Augmentation-Wise Weight Sharing (AWS) to create a fast yet accurate proxy task.
result Augmentation policies found achieve superior accuracies compared to existing methods.
Proposes a method to create robust linear models with noisy proxies of unobserved variables.
problem Learning robust linear models to handle interventions on unobserved variables with noisy proxies.
method Regularization term that balances in-distribution performance and robustness to interventions.
result Single proxy can create prediction optimal estimators under interventions of bounded strength.
One of the biggest bottlenecks in a machine learning workflow is waiting for models to train. Depending on the available computing resources, it can take days to weeks to train a neural network on a large dataset with many classes such as ImageNet. For researchers experimenting with new algorithmic approaches, this is …
Bayesian method estimates causal effects with proxy networks.
problem Estimating causal effects with only proxy measurements of a latent interference network.
method Structural causal model with Block Gibbs sampler and Locally Informed Proposals.
result Accurately estimates causal effects even with noisy proxy networks.
Kernel methods estimate causal effects with a single proxy for deterministic confounders.
problem Estimating causal effects with a single proxy for an unobserved confounder.
method Two kernel-based methods: two-stage regression and maximum moment restriction.
result Both kernel methods can consistently estimate the causal effect.