Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
Survey of Optimal Transport for model calibration.
problem Model calibration using Optimal Transport.
method General framework and numerical algorithms for various models.
result Calibration of volatility models and path-dependent options.
This post introduces model calibration and evaluation measures, highlighting issues with a common measure.
problem Ensuring model confidence accurately reflects true outcomes.
method Explains common calibration definition, ECE, and its drawbacks.
result New evaluation measures needed for comprehensive model calibration.
Certified calibration methods protect model confidence from adversarial attacks.
problem Adversarial attacks degrade model calibration, reducing confidence in predictions.
method Developed certified calibration methods to provide worst-case bounds on calibration under adversarial perturbations.
result Certified calibration methods produce analytic and approximate bounds for the Brier score and expected calibration error.
Improves calibration of regression models without requiring additional data.
problem Poor calibration of regression models leading to unreliable predictions.
method Quantile regularizer based on cumulative KL divergence.
result Significantly improves calibration for regression models trained with Dropout VI and Deep Ensembles.
We propose a new framework to improve the calibration of neural networks.
problem Improving the accuracy of model confidence predictions.
method Introducing a differentiable surrogate for expected calibration error (DECE) and a meta-learning framework to optimise model hyper-parameters for validation set calibration.
result Achieved competitive performance with existing calibration approaches.
The paper addresses poor calibration in fine-tuned LLMs after preference alignment.
problem Poor calibration in fine-tuned Large Language Models (LLMs) after preference alignment.
method Proposes a calibration-aware fine-tuning approach to restore calibration without compromising model performance.
result Demonstrates the effectiveness of the proposed methods through extensive experiments.
Develops confidence intervals for ECE, a measure of model calibration.
problem Ensuring the calibration of probabilistic predictions in machine learning models.
method Develops confidence intervals for the ℓ2 Expected Calibration Error (ECE), considering top-1-to-k calibration. result Shows asymptotic normality and different convergence rates for calibrated and miscalibrated models, developing methods to construct valid confidence intervals.
Efficiently calibrates epidemiological models using Bayesian optimization.
problem Calibration of complex epidemiological models is computationally expensive and challenging.
method Graybox Bayesian optimization scheme leveraging Gaussian processes and functional structure of compartmental models.
result Proposed methods achieve efficient calibration and improved performance compared to existing schemes.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
New study on neural network calibration, linking it to generalization gap.
problem Neural networks lack strong guarantees on calibration.
method Decomposed calibration error into train set and generalization gap.
result Models with small generalization gap are well-calibrated.
Time series foundation models are well-calibrated, improving over baseline models.
problem Calibration of time series foundation models for practical applications.
method Systematic evaluations of five time series foundation models and two baselines, assessing calibration, prediction heads, and long-term forecasting.
result Time series foundation models are consistently better calibrated than baseline models and do not show over- or under-confidence.
THERMOMETER calibrates LLMs efficiently for diverse tasks.
problem Calibrating large language models is challenging due to computational and versatility issues.
method THERMOMETER learns an auxiliary model for calibrating a LLM using data from multiple tasks.
result THERMOMETER produces better-calibrated responses for new tasks.
Survey of methods to calibrate neural network predictions.
problem Ensuring neural networks provide accurate confidence levels.
method Empirical comparison of calibration methods.
result Various techniques for calibrating neural networks.
T-Cal tests model calibration with a minimax optimal test.
problem Detecting mis-calibration of predictive models using a finite validation dataset.
method T-Cal is a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ2-Expected Calibration Error (ECE). result T-Cal is a practical tool for testing the calibration of probabilistic classification methods.
New method turns any regression model into a calibrated probabilistic model.
problem Calibration and sharpness of uncertainty estimates in regression models.
method Modular Conformal Calibration (MCC) framework.
result MCC algorithms achieve near-perfect calibration and improved sharpness.
A new metric CKCE improves model calibration comparison.
problem Comparing the calibration of probabilistic models is challenging.
method CKCE based on Hilbert-Schmidt norm of conditional mean operators.
result CKCE provides more consistent and robust model calibration comparisons.
New truthful calibration errors improve model ranking in multiclass prediction.
problem Non-truthful calibration errors can mislead model comparisons.
method Introduced perfectly truthful calibration errors for multiclass predictions.
result Truthful calibration errors preserve decision-theoretic dominance and stabilize model rankings.
X-CAL improves survival model calibration without sacrificing predictive power.
problem Improving the calibration of survival models to better match observed data.
method Explicit calibration (X-CAL) turns distributional calibration into a differentiable objective for survival modeling.
result X-CAL improves calibration metrics without significantly reducing predictive performance.
Self-calibrating conformal prediction improves interval efficiency and offers a practical alternative.
problem Improving the reliability and uncertainty quantification of machine learning predictions.
method Combines Venn-Abers calibration and conformal prediction for binary and regression problems.
result Improves interval efficiency through model calibration and offers practical alternatives.
Reassesses calibration metrics in machine learning models.
problem Inconsistent reporting of calibration metrics in recent literature.
method Calibration-based decomposition of Bregman divergences, visualization of calibration and generalization error.
result New visualization technique for detecting trade-offs between calibration and generalization.
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.
Improves model calibration for deep neural networks using proper scores.
problem Calibration errors in deep neural networks are often biased and inconsistent.
method Introduces proper calibration errors related to proper scores.
result Demonstrates the superiority of proper scores over common estimators.
MCNet improves uncertainty calibration in online advertising by modeling complex relations and balancing performance.
problem Lack of effective calibration for complex relations and context features in online advertising.
method Introduces MCNet with MCF, order-preserving, and field-balance regularizers.
result Superior performance in generating well-calibrated probability predictions on public and industrial datasets.
Efficiently calibrates Bergomi models to VIX derivatives using vector quantization.
problem Calibrating Bergomi models to VIX derivatives for accurate pricing.
method Applied vector quantization in mixed Bergomi models for fast and efficient option pricing.
result Calibration of Bergomi models to VIX derivatives is feasible and accurate over daily data.
New method improves calibration in multi-output probabilistic models.
problem Challenges in achieving multivariate calibration in multi-output regression.
method General regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions.
result Significant improvement in calibration across all pre-rank functions without sacrificing predictive accuracy.
SVI and GP surrogates improve calibration of ABMs in epidemiology.
problem Calibrating stochastic ABMs in epidemiology is computationally expensive.
method Stein Variational Inference (SVI) with Gaussian process (GP) surrogates.
result SVI maintains comparable predictive accuracy and calibration effectiveness to MCMC.
Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popu…
Paper proposes a unified framework for evaluating calibration of probabilistic models.
problem Evaluation of calibration for general probabilistic predictive models.
method Unified framework for calibration evaluation and tests for any probabilistic model.
result Generalization and reformulation of existing measures and tests.
Boosting trees can test necessary conditions for regression model calibration.
problem Testing calibration and auto-calibration in regression models.
method Using boosting trees to test calibration and auto-calibration.
result Boosting trees prove to be very powerful in testing calibration and auto-calibration in large insurance datasets.
Proposes an accuracy-preserving calibration method for DNNs.
problem Calibration of deep neural networks (DNNs) to measure prediction reliability.
method Uses Concrete distribution on the probability simplex to calibrate DNNs without accuracy loss.
result The proposed method outperforms previous methods in accuracy-preserving calibration tasks.
Method calibrates local volatility and stochastic short rate models for equity-rate dynamics.
problem Joint calibration of local volatility and stochastic short rate models.
method Iterative approach using semimartingale optimal transport.
result Demonstrated performance on market data using European SPX options and cap interest rate options.
New method calibrates models under covariate shifts.
problem Calibration of models can be lost under covariate shifts.
method Importance sampling based approach.
result Efficacy demonstrated on real-world and synthetic datasets.
We study statistical calibration, i.e., adjusting features of a computational model that are not observable or controllable in its associated physical system. We focus on functional calibration, which arises in many manufacturing processes where the unobservable features, called calibration variables, are a function of…
Human-AI teaming suffers from calibration issues.
problem Human-AI teaming
method Assume calibrated models and humans
result Existing methods for combination do not preserve human's calibration.
CalArena benchmarks post-hoc calibration methods across various tasks.
problem Inconsistent evaluations of post-hoc calibration methods.
method Large-scale benchmark with 2000 experiments, covering diverse models and settings.
result Smooth calibration functions outperform binning-based approaches.
We propose nonparametric methods for individual calibration in regression models.
problem Uncertainty quantification and individual calibration for regression models.
method Nonparametric methods agnostic of the underlying model, combining nonparametric and covering number arguments.
result Established matching upper and lower bounds for calibration error.
Obtaining accurate and well calibrated probability estimates from classifiers is useful in many applications, for example, when minimising the expected cost of classifications. Existing methods of calibrating probability estimates are applied globally, ignoring the potential for improvements by applying a more fine-gra…
New method calibrates heterogeneous treatment effect models.
problem Difficulty in estimating and calibrating heterogeneous treatment effects.
method Defined and proposed a robust estimator for HTE calibration, based on doubly robust treatment effect estimators.
result Proposed method evaluates calibration of learned HTE models, addressing overfitting and high-dimensionality.
Generative models often misrepresent class frequencies; this paper calibrates them.
problem Miscalibration of class frequencies in generative models.
method Formulated as constrained optimization, using surrogate objectives to approximate constraints.
result Significant reduction in calibration error across various models and applications.
In this paper we develop a tractable structural model with analytical default probabilities depending on some dynamics parameters, and we show how to calibrate the model using a chosen number of Credit Default Swap (CDS) market quotes. We essentially show how to use structural models with a calibration capability that …
This work addresses building fair and calibrated models.
problem Building models that are both fair and calibrated.
method Developed a new definition of fairness and showed that group-wise calibration results in fairness. Proposed post-processing techniques and modifications of calibration losses.
result Demonstrated that ensuring group-wise calibration results in a fair model under the new definition of fairness.
In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class…
The paper proposes a new method to calibrate multiple computer models simultaneously.
problem Calibrating multiple computer models one at a time is inefficient.
method Developed a probabilistic framework using customized neural networks.
result Simultaneous calibration improves predictive accuracy but can be non-identifiable in high dimensions.
Combining ensembles and data augmentation harms model calibration.
problem Improving model calibration and robustness with ensembles and data augmentation leads to a trade-off.
method Combining ensemble averaging and data augmentation techniques.
result Combining ensembles and data augmentation can harm model calibration.
Space mapping calibrates financial models, shown feasible for Heston model.
problem Calibrating financial models with few observable parameters and non-linear constraints.
method Space mapping approach using a coarse surrogate model and fine model calibration.
result Space mapping approach feasible for Heston model calibration.
Improves forecast calibration for extreme events using modified loss functions.
problem Improperly specified models do not issue calibrated forecasts for extreme events.
method Adapting loss functions based on weighted scoring rules and tail miscalibration regularization.
result Calibrated forecasts for extreme wind speeds can be improved by suitable adaptations to the loss function during model training.
Study improves neural network calibration for drug discovery.
problem Improper calibration of neural network predictions in drug discovery.
method Compared different metrics for model hyperparameter tuning and proposed Bayesian Linear Probing (BLP) method.
result Bayesian Linear Probing (BLP) improves model calibration and accuracy.