Unified calibration metrics improve forecast sharpness and accuracy.
problem Improving the sharpness of probabilistic forecasts while maintaining calibration.
method Kernel-based calibration metrics that unify and generalize existing methods for classification and regression.
result Enhanced calibration, sharpness, and decision-making across various tasks.
Proposes h-calibration for improving miscalibrated probability outputs of neural networks.
problem Improving reliability of probability outputs from neural networks.
method Probabilistic learning framework for calibration, including a simple yet effective post-hoc algorithm.
result Significantly better performance than traditional methods, validated by experiments.
This paper introduces a novel recalibration method for multivariate forecasts.
problem Multivariate calibration for potentially misspecified models.
method Local mappings between marginal probability integral transform values and observed space, using K-nearest neighbors or normalizing flows.
result Demonstrated effectiveness on currency exchange rate and childhood malnutrition data.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
End-to-end method improves neural network calibration during training.
problem Improving neural network calibration for regression problems.
method Quantile Recalibration Training integrates post-hoc calibration into model training.
result Improved predictive accuracy and calibration in a large-scale experiment.
This work proves L2-regularized ERM controls smCE without post-hoc correction.
problem Calibration of predicted probabilities in machine learning models.
method Canonical L2-regularized empirical risk minimization. result Theoretical proof that smCE is controlled by ERM without post-hoc correction.
Study evaluates uncertainty in BP estimation from PPG signals under domain shift.
problem Uncertainty quantification in healthcare, especially for cuffless BP estimation.
method Compared deep ensembles, Monte Carlo dropout, and various recalibration techniques.
result Deep ensembles provide stronger robustness under domain shift.
Isotonic regression binning affects calibration statistics of machine learning models.
problem Isotonic regression binning introduces aleatoric uncertainty in calibration statistics.
method Calibration error statistics are recalibrated using isotonic regression, which produces stratified uncertainties.
result Stratified uncertainties lead to significant differences in bin-based calibration statistics.
Unified approach for multicalibration in weakly supervised learning.
problem Existing multicalibration methods require clean input-label pairs, which are unavailable in weakly supervised learning.
method Developed estimators and post-hoc correction methods for multicalibration under weak supervision.
result Unified framework for estimating and correcting multicalibration under weak supervision with finite-sample guarantees.
Proposes isotonic recalibration for insurance pricing to ensure auto-calibration under low signal-to-noise ratio.
problem Ensuring auto-calibration in insurance pricing models to prevent cross-financing.
method Applies isotonic recalibration to regression models to achieve auto-calibration.
result Isotonically recalibrated regression functions have low complexity under low signal-to-noise ratio.
This paper introduces minimum-risk recalibration for probabilistic classifiers, improving their reliability and accuracy.
problem Improving the reliability and accuracy of probabilistic classifiers.
method Minimum-risk recalibration within the MSE decomposition framework, analyzing UMB method and label shift adaptation.
result The optimal number of bins for UMB scales with n1/3, resulting in a risk bound of approximately O(n−2/3). Framework improves classifier calibration under differential privacy for domain shift.
problem Improving classifier calibration under domain shift with privacy constraints.
method Differential privacy framework for adapting recalibration algorithms.
result Novel accuracy temperature scaling algorithm outperforms existing methods on private datasets.
New bin-wise scaling methods improve prediction uncertainty calibration for machine learning.
problem Improving prediction uncertainty calibration for machine learning regression.
method Adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and feature-based binning.
result Improved adaptivity and consistency in prediction uncertainty calibration.
The paper analyzes recalibration methods for binary classifiers under distribution shift.
problem Recalibrating binary classifiers to match a target prior probability.
method Analysis of distribution shift assumptions and proposal of new recalibration methods.
result QMM methods provide conservative results for risk weights functions.
Optimizes predictions by recalibrating online forecasts with minimal error.
problem Tackles the challenge of recalibrating online predictions to be more accurate.
method Uses an imbalanced extension of the Blackwell approachability reduction framework to achieve (ε,ε2)-recalibration. result Achieves (ε,ε2)-recalibration for Lipschitz proper losses in T≈ε−3 rounds. A new framework improves VaR recalibration by balancing reliance on imperfect volatility proxies.
problem How to balance reliance on imperfect volatility proxies in one-sided VaR recalibration.
method Proxy-reliance control framework that interpolates between constant-shift and proxy-scaled corrections.
result Lower or intermediate proxy reliance can outperform fully proxy-scaled recalibration in stressed left-tail VaR control.
PE-GQNN improves spatial data prediction and uncertainty quantification.
problem Poor calibration of predictive distributions in spatial data models.
method Combines PE-GNNs with Quantile Neural Networks and recalibration techniques.
result PE-GQNN outperforms existing methods in predictive accuracy and uncertainty quantification.
New method calibrates machine learning models with theoretical guarantees.
problem Lack of theoretical guarantees for recalibration in multiclass classification.
method PAC-Bayes analysis for generalization error in calibration.
result First optimizable upper bound for generalization error in calibration.
New method recalibrates VaR for option books, reducing forecast errors.
problem Inaccurate VaR forecasts due to missing operational choices.
method Marking-aware sequential VaR recalibration targeting normalized book-level loss.
result Sequential VaR recalibration improves VaR performance across different markets and options.
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
Proposes MCLLO for assessing and recalibrating multiclass probability predictions.
problem Limited multicategory recalibration methods for assessing and comparing model calibration.
method MCLLO recalibration method that assesses calibration without model access and is easy to interpret.
result MCLLO outperforms other methods in simulations and real-world case studies.
3D ConvNets improved with Project & Excite for medical imaging segmentation.
problem Improving segmentation performance in 3D medical imaging.
method Proposed Project & Excite (PE) modules for 3D F-CNNs, extending 2D recalibration methods.
result Project & Excite modules boost segmentation performance up to 0.3 in Dice Score.
TransCal calibrates DA models with lower bias and variance.
problem Calibrating DA models to estimate accurate predictive uncertainty.
method Transferable Calibration (TransCal) in a unified hyperparameter-free optimization framework.
result TransCal achieves more accurate calibration with lower bias and variance.
This paper addresses recalibration issues in hedging callable assets, proposing a new risk-adjusted approach.
problem The mismatch between dynamic hedging theory and practice due to daily recalibration.
method Extends HVA model risk approach to callable assets, focusing on recalibration and model risks.
result Model risk reserves adjusted for exercise decisions may significantly exceed basic valuation differences.
New models capture dynamic derivatives pricing with efficient simulations.
problem Capturing dynamic features of derivatives' term structures.
method Machine learning techniques to store and efficiently simulate complex drift terms.
result First efficient dynamic term structure models.
A new method calibrates value predictions in offline RL to improve reliability.
problem Difficulty in long-horizon value prediction in offline reinforcement learning.
method Bellman calibration, a weak reliability criterion, and Iterated Bellman Calibration.
result Finite-sample guarantees show that Bellman calibration error is controlled at nonparametric rates.
We focus on two particular aspects of model risk: the inability of a chosen model to fit observed market prices at a given point in time (calibration error) and the model risk due to recalibration of model parameters (in contradiction to the model assumptions). In this context, we follow the approach of Glasserman and …
The paper enhances representations to show left-orderability of certain 3-manifold groups.
problem Left-orderability of 3-manifold groups using enhanced representations.
method Recalibration of Calegari and Dunfield's flipping construction for $\mbox{Homeo}_+(S^1)$-representations.
result Branched covers of links are left-orderable, generalizing known results.
Proposes a method to generate multivariate prediction intervals for random forests.
problem Uncertainty estimates for iterative design of experiments with multiple correlated model outputs.
method Recalibrated bootstrap method for bagged models.
result Significantly decreases the number of iterations required for satisfactory candidate in sequential learning problems.
We study option pricing and hedging with uncertainty about a Black-Scholes reference model which is dynamically recalibrated to the market price of a liquidly traded vanilla option. For dynamic trading in the underlying asset and this vanilla option, delta-vega hedging is asymptotically optimal in the limit for small u…
Post-hoc transforms can reverse model performance trends, especially in noisy settings.
problem Post-hoc transforms can reverse model performance trends, especially in noisy settings.
method Empirical study and analysis of post-hoc transforms like temperature scaling, ensembling, and SWA.
result Post-hoc reversal can prevent double descent and mitigate mismatches between test loss and test error.
This work evaluates and benchmarks calibration metrics for data-driven regression models.
problem Conflicting results from different calibration metrics make it hard to compare and interpret model performance.
method Systematically extracted and benchmarked 14 regression calibration metrics across various data types and recalibration methods.
result Many metrics disagree on the same recalibration result, highlighting the need for careful metric selection.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Tabular foundation models outperform other methods in conditional density estimation across various datasets.
problem Estimating the full conditional distribution of a response given tabular covariates, especially in settings with heteroscedasticity, multimodality, or asymmetric uncertainty.
method Benchmarked three tabular foundation model variants (TabPFN and TabICL) against six CDE baselines on 39 real-world datasets.
result Tabular foundation models achieve the best CDE loss, log-likelihood, and CRPS across all sample sizes, outperforming other methods.
Framework evaluates post-hoc interpretability methods in time-series classification.
problem Lack of suitable post-hoc interpretability methods for time-series classification.
method Proposes a framework with quantitative metrics to assess interpretability methods.
result Addresses several drawbacks of existing methods, including dependence on human judgement and data distribution shift.
The paper introduces a spline-based method for calibrating neural networks.
problem Ensuring neural network outputs are reliable for safety-critical applications.
method Approximating the empirical cumulative distribution function using splines to map network outputs to calibrated probabilities.
result The spline-based recalibration consistently outperforms existing methods on calibration measures.
Mathematical study shows post-hoc explanations are better than attention weights alone.
problem Understanding the internal behavior of attention-based models.
method Mathematical analysis of a simple attention-based architecture.
result Post-hoc explanations provide more useful insights than attention weights alone.
New method turns any regression model into a calibrated probabilistic model.
problem Calibration and sharpness of uncertainty estimates in regression models.
method Modular Conformal Calibration (MCC) framework.
result MCC algorithms achieve near-perfect calibration and improved sharpness.
CalArena benchmarks post-hoc calibration methods across various tasks.
problem Inconsistent evaluations of post-hoc calibration methods.
method Large-scale benchmark with 2000 experiments, covering diverse models and settings.
result Smooth calibration functions outperform binning-based approaches.
Post hoc test for Sharpe ratio improves pairwise comparisons.
problem Improving pairwise comparisons of Sharpe ratios.
method Analogous to Tukey's test, applied after rejecting equal Signal-Noise ratios.
result Maintains nominal type I rate and is moderately powerful.
The paper calibrates uncertainty in dropout variational inference models.
problem Miscalibration of model uncertainty in dropout variational inference.
method Logit scaling methods are extended to recalibrate model uncertainty.
result Logit scaling reduces miscalibration, improving reliability of predictions.
Study post-hoc Learning to Defer using density-ratio losses.
problem Optimizing decision-making between models and experts.
method Density-ratio losses for post-hoc L2D scorers, derived from class-probability estimation.
result The approach recovers known results and introduces new connections to expert comparison and anomaly detection.
A new framework for PPLS combines noise estimation, optimization, and calibration.
problem Probabilistic PLS models need interpretable latent factors and calibrated uncertainty.
method End-to-end pipeline combining noise estimation, constrained optimization, and prediction calibration.
result Achieves near-nominal coverage and native calibrated uncertainty across benchmarks.
Post-hoc calibration of neural networks using g-Layers proves theoretical justification.
problem Ensuring the confidence of neural network decisions in real-world applications.
method Proves theoretical justification for post-hoc calibration methods by adding g-Layers and minimizing NLL.
result Proves that adding g-Layers and minimizing NLL can lead to a calibrated network.
A post-hoc framework improves model performance by calibrating different feature spaces.
problem Improving AUC performance on binary classification tasks for overconfident models.
method Identifies heterogeneous partitions of the feature space and applies post-hoc calibration techniques to each partition.
result Theoretical optimality of the framework for any model, demonstrated on deep neural networks.
Counterfactual post-hoc interpretability approaches have been proven to be useful tools to generate explanations for the predictions of a trained blackbox classifier. However, the assumptions they make about the data and the classifier make them unreliable in many contexts. In this paper, we discuss three desirable pro…
The role of uncertainty quantification (UQ) in deep learning has become crucial with growing use of predictive models in high-risk applications. Though a large class of methods exists for measuring deep uncertainties, in practice, the resulting estimates are found to be poorly calibrated, thus making it challenging to …
New method uses AI predictions as cheaper alternatives to expensive outcomes.
problem Using expensive outcomes for statistical inference.
method Recalibrated prediction-powered inference using machine learning techniques.
result Significant gains in effective sample size over existing PPI proposals.