Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
Paper proposes a deep RL method for hedging variable annuities, outperforming misspecified models.
problem Model miscalibration in variable annuity contracts with GMMB and GMDB riders.
method Two-phase deep reinforcement learning approach: training phase in a controlled environment, online learning phase in real market.
result Trained reinforcement learning agent hedges equally well as correct Delta in training phase and outperforms misspecified Deltas.
Statistical inference for misspecified contextual bandits is challenging due to adaptivity issues.
problem Statistical inference for misspecified contextual bandits
method Inverse-probability-weighted Z-estimation framework
result Consistent and asymptotically normal estimator with sandwich variance estimator
Bayesian algorithms perform well even with misspecified priors, especially in meta-learning.
problem Performance degradation of Bayesian algorithms with misspecified priors.
method Thompson sampling and meta-learning analysis with misspecified priors.
result Thompson sampling's performance degrades gracefully with misspecification, with a bound of i l d e O ( H 2 ε ) ilde{\mathcal{O}}(H^2 ε) i l d e O ( H 2 ε ) . New algorithm optimizes smooth functions with Hölder exponent > 1.
problem Optimizing smooth functions with unknown Hölder exponent > 1.
method Two-layer algorithms using misspecified linear/polynomial bandit algorithms in bins.
result Regret bound of O ~ ( T d + α d + 2 α ) \tilde{O}(T^{\frac{d+\alpha}{d+2\alpha}}) O ~ ( T d + 2 α d + α ) for α > 1 \alpha > 1 α > 1 . This paper explores how imperfect reward models can improve online RLHF.
problem Sample efficiency in online RLHF from imperfect reward models.
method Identifies policy coverability and proposes TPO algorithm for transfer learning.
result TPO algorithm improves sample efficiency compared to standard online learning.
Adaptive inference for M M M -estimators in bandit data with model misspecification.
problem Challenges in off-policy inference for adaptively collected bandit data with a misspecified model.
method A novel approach to define a projected solution over a stationary evaluation policy, stabilizing variance with flexible methods.
result Valid inference for M M M -estimators in adaptive settings, even with unstable treatment policies. The paper addresses statistical inference for online decision-making in a contextual bandit setting.
problem Understanding the performance of reward models in online decision-making with contextual information.
method The paper uses the contextual bandit framework with a linear reward model and the ε \varepsilon ε -greedy policy to address the exploration-exploitation dilemma. It employs the martingale central limit theorem and inverse propensity score weighting to establish asymptotic normality of parameter estimators. result The online ordinary least squares estimator and the online weighted least squares estimator are asymptotically normal, providing insights into the performance of the reward model.
Optimizes forecast distributions for financial risk management.
problem Improving risk management through better forecast distributions.
method Optimizes forecast distributions using scoring rules relevant to financial risk management.
result Tail-focused predictive distributions yield better outcomes in hedging strategies involving VIX futures.
The paper examines when importance weighting is needed for nonparametric and misspecified models.
problem When is importance weighting correction needed for covariate shift adaptation?
method Analysis of IW-corrected kernel ridge regression in various settings.
result The importance weighting correction is needed for nonparametric and misspecified models to obtain the best approximation of the true unknown function.
The paper examines Gaussian process means under misspecified likelihoods and smoothness.
problem Accuracy of Gaussian process approximations under misspecified smoothness and likelihood.
method Analysis of Gaussian process properties under misspecified conditions.
result The accuracy of Gaussian process approximations is influenced by experimental design and kernel choice.
Bayesian bandits misspecification affects UX optimization, revealing new models.
problem Misspecification of value models in Bayesian bandits impacts UX optimization.
method Formulated UXO as a restless, sleeping bandit with unobserved confounders and optional stopping. Provided model extensions to address misspecifications.
result Common misspecifications lead to sub-optimal rewards, demonstrating overdispersion's effects on bandit performance.
New PG losses improve decision optimization in misspecified models.
problem Improving decision optimization in models that are not perfectly specified.
method Introducing Perturbation Gradient (PG) losses to connect decision loss with directional derivatives and optimizing using gradient techniques.
result PG losses yield best-in-class policies asymptotically, even in misspecified settings.
Algorithm reduces regret in misspecified linear contextual bandits.
problem Misspecified linear contextual bandits with bounded misspecification.
method Data selection scheme for online regression, leveraging uncertainty.
result Regret bound of O ~ ( d 2 / Δ ) \tilde O(d^2/Δ) O ~ ( d 2 /Δ ) when ζ ≤ O ~ ( Δ / d ) ζ \leq \tilde O(Δ/\sqrt{d}) ζ ≤ O ~ ( Δ/ d ) . Proposes m-POT to improve m-OT's misspecified mappings issue.
problem Misspecified mappings in mini-batch optimal transport.
method Partial optimal transport (POT) between mini-batch empirical measures.
result m-POT alleviates incorrect mappings compared to current methods.
Nyström subsampling with Tikhonov regularization for covariate shift adaptation under misspecified case
problem Adaptation to misspecified covariate shift
method Regularized Nyström subsampling with Tikhonov regularization
result Upper bounds on excess risk
New methods for CI testing under model misspecification.
problem Challenges in CI testing with misspecified models.
method Proposes new approximations and upper bounds for testing errors of regression-based CI tests.
result Introduces the Rao-Blackwellized Predictor Test (RBPT) robust against misspecified inductive biases.
Study on sequential prediction with log-loss, focusing on well-specified and misspecified cases.
problem Sequential prediction with log-loss under different specification conditions.
method Analysis of cumulative regret in well-specified and misspecified cases for a Gaussian location hypothesis class.
result Cumulative regrets in well-specified and misspecified cases asymptotically coincide for the d d d -dimensional Gaussian location hypothesis class. Study non-asymptotic bounds for robust estimators under misspecified models.
problem Evaluate performance of robust estimators under adversarial conditions.
method Propose a general approach to adversarial risk analysis, including investigations on generalization and approximation errors.
result Establish non-asymptotic upper bounds for adversarial excess risk under Lipschitz loss functions.
Suppose an investor aims at Delta hedging a European contingent claim h ( S ( T ) ) h(S(T)) h ( S ( T )) in a jump-diffusion model, but incorrectly specifies the stock price's volatility and jump sensitivity, so that any hedging strategy is calculated under a misspecified model. When does the erroneously computed strategy super-replicate the t…
Improved algorithm for misspecified MLMDPs with bounded regret and space/time complexities.
problem Misspecified linear Markov decision processes.
method Proposes an algorithm with three desirable properties: bounded regret, bounded space/time complexities, and no need for misspecification input.
result Regret scales as K max { ε e x t m i s , ε e x t t o l } K \max \{ \varepsilon_{ ext{mis}}, \varepsilon_{ ext{tol}} \} K max { ε e x t mi s , ε e x t t o l } , improving existing bounds. This paper presents a convergence analysis of kernel-based quadrature rules in misspecified settings, focusing on deterministic quadrature in Sobolev spaces. In particular, we deal with misspecified settings where a test integrand is less smooth than a Sobolev RKHS based on which a quadrature rule is constructed. We pr…
New method improves GP uncertainty quantification for misspecified priors.
problem Uncertainty quantification for GPs under incorrect priors.
method Constructs a confidence sequence using martingale techniques.
result Empirically outperforms standard GP methods in robustness and utility for Bayesian Optimization.
Improves off-policy evaluation with imperfect annotations.
problem Limited dataset coverage for evaluating new policies.
method Doubly robust estimators combining IS and DM, incorporating counterfactual annotations.
result Using annotations within the DM component yields the most desirable theoretical results.
Framework predicts responses in misspecified systems using GPLFM and BNNs.
problem Predicting responses in dynamical systems with model misspecification.
method Integrates GPLFM and BNNs for uncertainty-aware inference and prediction.
result Systematic propagation of uncertainty from diagnosis to prediction.
New RL method learns to skip states in linearly q π q^π q π -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly q π q^π q π -realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly q π q^π q π -realizable MDPs. Self-consistency improves the accuracy of model comparison methods.
problem Improving the accuracy of model comparison methods when simulation models are misspecified.
method Supplement traditional simulation-based training with a self-consistency loss on unlabeled real data.
result Self-consistency training improves model comparison accuracy, especially in open-world scenarios.
We consider a class of misspecified dynamical models where the governing term is only approximately known. Under the assumption that observations of the system's evolution are accessible for various initial conditions, our goal is to infer a non-parametric correction to the misspecified driving term such as to faithful…
The paper improves bandit algorithms by incorporating random-effect models.
problem Improving statistical efficiency in multi-armed bandit problems with misspecified priors.
method Introduces a random-effect model to bandits, estimating arm means and designing a UCB algorithm ReUCB.
result Derives an upper bound on the Bayes regret of ReUCB, showing improved performance over Thompson sampling.
Researchers compare lower bounds and tractability in reinforcement learning.
problem Intractability of reinforcement learning with misspecified representations.
method Comparison of lower bounds and eluder dimension approach.
result Reconciliation of interpretations between different lines of work.
Improved classifier for PU data using logistic regression.
problem Analysis of Positive Unlabeled data under SCAR assumption.
method Fitting misspecified logistic regression model to PU data.
result The classifier performs on par or better than competitors on real data sets.
The paper studies the robust maximization of utility of terminal wealth in the diffusion financial market model. The underlying model consists with risky tradable asset, whose price is described by diffusion process with misspecified trend and volatility coefficients, and non-tradable asset with a known parameter. The …
New algorithms for optimizing functions with noisy feedback, even when the model is misspecified.
problem Optimizing a black-box function with noisy bandit feedback, especially when the model is misspecified.
method Developed two algorithms based on Gaussian process methods: EC-GP-UCB and Phased GP Uncertainty Sampling.
result Achieved optimal dependence on misspecification error without prior knowledge, and effective in stochastic contextual settings.
Paper analyzes spectral algorithms under covariate shift, providing convergence rates.
problem Addressing distributional mismatch in regression models.
method Incorporates importance weights into spectral algorithms in RKHS.
result Establishes minimax-optimal convergence rates for misspecified cases.
RPPs improve deep learning models with soft equivariance constraints.
problem Balancing expressiveness and inductive biases in deep learning.
method Introducing Residual Pathway Priors (RPPs) to convert hard constraints into soft priors.
result RPPs enable models to learn structured solutions while retaining flexibility.
Existing nonconvex statistical optimization theory and methods crucially rely on the correct specification of the underlying "true" statistical models. To address this issue, we take a first step towards taming model misspecification by studying the high-dimensional sparse phase retrieval problem with misspecified link…
Efficient algorithms for low-rank bandits using subspace recovery.
problem Contextual bandits with low-rank reward matrices.
method Spectral methods for subspace recovery, reformulating as linear bandits.
result Nearly optimal policy evaluation and best policy identification, minimax guarantees for regret minimization.
New algorithms for best arm identification in bandits robust to misspecified parameters.
problem Inconsistent learning performance of traditional MAB algorithms when parameters are misspecified.
method Proposes two classes of asymptotically near-optimal algorithms for statistically robust MAB under fixed-budget pure exploration.
result Establishes fundamental performance limits and proposes algorithms that are asymptotically near-optimal.
There is vast empirical evidence that given a set of assumptions on the real-world dynamics of an asset, the European options on this asset are not efficiently priced in options markets, giving rise to arbitrage opportunities. We study these opportunities in a generic stochastic volatility model and exhibit the strateg…
Improves Bayesian predictive performance in misspecified models.
problem Misspecification gap between inferential and predictive risks.
method Develops a multi-sample loss (PAC m ^m m ) to bridge the gap. result Empirical study shows improved predictive distribution.
New method improves uncertainty quantification for large batch sizes and misspecified models.
problem Challenges in tuning algorithms for accurate uncertainty quantification in large batch sizes and misspecified models.
method Proposes new discrete-time approximations to SGD and SGLD, proving error bounds for practical tuning.
result Quantitative, non-asymptotic error bounds for accurate predictions of covariance and autocorrelation time.
RoPE framework calibrates misspecified simulators for reliable inference.
problem Misspecification compromises reliability of simulation-based inference.
method Data-driven calibration using optimal transport and a small calibration set.
result RoPE framework improves inference accuracy and uncertainty calibration.
Sharp bounds on ATE with unmeasured confounders, valid even when misspecified.
problem Bounding average treatment effects with unmeasured confounders.
method Distributionally robust optimization, double sharpness, double validity.
result Proposes estimators with robustness properties for valid bounds.
Bayesian metalearning improves performance in linear bandits with misspecified priors.
problem Improper priors lead to suboptimal performance in sequential decision-making.
method Proves performance bounds for metalearning priors in stochastic linear bandits and develops a metalearning algorithm.
result Metalearning can improve performance by learning the prior from multiple tasks.
Plug-in method improves performative prediction accuracy.
problem Learning under performative feedback with slow convergence rates.
method Plug-in performative optimization using models.
result Plug-in method can be superior to model-agnostic strategies.
A new method improves efficiency in finding optimal personalized treatment rules.
problem Heteroscedasticity and misspecified treatment-free effect models affect optimal ITR estimation.
method E-Learning framework that accounts for covariate-treatment dependent variance of residuals.
result E-Learning framework improves efficiency of optimal ITR estimation.
Paper develops a method to predict spatial point processes with guarantees.
problem Predicting the number of events in space with uncertainty.
method Regularized method to learn spatial models with out-of-sample guarantees.
result Method provides valid prediction intervals even when model is misspecified.
ACE improves GBI for simulators by approximating cost functions, making inference more efficient.
problem Inference for misspecified simulators is overly restrictive.
method Amortized cost estimation (ACE) for Generalized Bayesian Inference (GBI).
result ACE provides accurate cost predictions and more efficient inference.