Proposes new method for calibrating treatment effect predictors.
problem Calibrating predictors of heterogeneous treatment effects.
method Causal isotonic calibration and cross-calibration.
result Achieves fast calibration rates under weak conditions.
Proposes top-label calibration and M2B framework for multiclass to binary calibration.
problem Multiclass calibration and interpretation issues.
method Top-label calibration and M2B reduction framework.
result M2B + HB achieves lower calibration error than other methods.
Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
A new perfectly truthful calibration measure improves prediction reliability.
problem Improving the reliability of predictions by ensuring they are conditionally unbiased.
method Designing a simple, perfectly truthful calibration measure called ATB.
result ATB is the first perfectly truthful calibration measure in the batch setting.
New truthful calibration errors improve model ranking in multiclass prediction.
problem Non-truthful calibration errors can mislead model comparisons.
method Introduced perfectly truthful calibration errors for multiclass predictions.
result Truthful calibration errors preserve decision-theoretic dominance and stabilize model rankings.
New framework for evaluating multiclass classifier calibration.
problem Ensuring classifiers are well-calibrated for trustworthy predictions.
method Utility Calibration framework that measures calibration error relative to a utility function.
result Unified and robust interpretation of existing calibration metrics.
We propose a new framework to improve the calibration of neural networks.
problem Improving the accuracy of model confidence predictions.
method Introducing a differentiable surrogate for expected calibration error (DECE) and a meta-learning framework to optimise model hyper-parameters for validation set calibration.
result Achieved competitive performance with existing calibration approaches.
Certified calibration methods protect model confidence from adversarial attacks.
problem Adversarial attacks degrade model calibration, reducing confidence in predictions.
method Developed certified calibration methods to provide worst-case bounds on calibration under adversarial perturbations.
result Certified calibration methods produce analytic and approximate bounds for the Brier score and expected calibration error.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
Post-processing predictors reduces calibration errors for decision-making.
problem Predictors with low calibration error for machine learning may have high error for decision-making.
method Post-processing with ε distance to calibration adds noise to make predictions differentially private.
result Post-processing achieves O(√ε) ECE and CDL, asymptotically optimal.
A new calibration metric bridges testability and actionability.
problem Combining testability and actionable insights for forecast probabilities.
method Cutoff Calibration Error (CCE) that assesses calibration over intervals of forecasted probabilities.
result Cutoff Calibration Error is both testable and actionable.
New study on neural network calibration, linking it to generalization gap.
problem Neural networks lack strong guarantees on calibration.
method Decomposed calibration error into train set and generalization gap.
result Models with small generalization gap are well-calibrated.
A new method for multiclass calibration using vector quantization.
problem Challenges in multiclass calibration, especially in high-stakes settings.
method Compositional approach via Vector Quantization (VQ) to learn region-specific calibration maps.
result Significant improvements in local calibration with competitive global calibration and predictive performance.
This post introduces model calibration and evaluation measures, highlighting issues with a common measure.
problem Ensuring model confidence accurately reflects true outcomes.
method Explains common calibration definition, ECE, and its drawbacks.
result New evaluation measures needed for comprehensive model calibration.
Unified calibration metrics improve forecast sharpness and accuracy.
problem Improving the sharpness of probabilistic forecasts while maintaining calibration.
method Kernel-based calibration metrics that unify and generalize existing methods for classification and regression.
result Enhanced calibration, sharpness, and decision-making across various tasks.
New decision-theoretic calibration error metric improves prediction reliability.
problem Improving the reliability of predictions for decision-making.
method Proposed Calibration Decision Loss (CDL) and an efficient algorithm to achieve near-optimal CDL.
result Near-optimal CDL guarantees vanishing payoff loss from miscalibration.
Cone structures over minimal products can't be calibrated smoothly.
problem Calibrating cones over minimal products with smooth calibrations.
method Extending a key result from [Zha26], showing obstruction.
result Cone structures over minimal products cannot be calibrated by smooth calibrations.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Smooth calibration improves forecast reliability even with leaked information.
problem Improving forecast reliability with leaked information.
method Combining nearby forecasts to ensure smooth calibration, which can be guaranteed by deterministic procedures.
result Smooth calibration can be guaranteed by deterministic procedures even with leaked forecasts, and it yields uncoupled finite-memory dynamics in games.
Planes are the only calibrated submanifolds with flat normal bundles.
problem Characterizing submanifolds with specific geometric properties.
method Using constant-coefficient differential forms and parallel calibrations.
result Calibrated submanifolds with flat normal bundles are planes.
We describe a family of calibrations arising naturally on a hyperkähler manifold M M M . These calibrations calibrate the holomorphic Lagrangian, holomorphic isotropic and holomorphic coisotropic subvarieties. When M M M is an HKT (hyperkaehler with torsion) manifold with holonomy S L ( n , H ) SL(n, {\Bbb H}) S L ( n , H ) , we construct another fam…
Simple proof shows forecasts can be calibrated in a few periods.
problem Ensuring forecasts are calibrated over multiple periods.
method Uses minimax theorem to prove existence and guarantees calibration error.
result Calibration can be achieved in N 3 N^3 N 3 periods with error at most 1 / N 1/N 1/ N . Survey of methods to calibrate neural network predictions.
problem Ensuring neural networks provide accurate confidence levels.
method Empirical comparison of calibration methods.
result Various techniques for calibrating neural networks.
This paper rethinks confidence calibration under covariate shifts.
problem Calibration methods struggle with covariate shifts and unstable importance weighting.
method Derives Expectation consistency condition and proposes Expectation consistency loss (ECL).
result ECL loss is compatible with various types of calibration and has the same sample complexity as ECE.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
New conditions for calibrated submanifolds in Riemannian geometry.
problem Characterizing calibrated submanifolds with extrinsic geometry.
method Introducing compliancy condition and analyzing extrinsic geometry.
result Conditions for extrinsic geometry of calibrated submanifolds.
Survey of Optimal Transport for model calibration.
problem Model calibration using Optimal Transport.
method General framework and numerical algorithms for various models.
result Calibration of volatility models and path-dependent options.
Develops confidence intervals for ECE, a measure of model calibration.
problem Ensuring the calibration of probabilistic predictions in machine learning models.
method Develops confidence intervals for the ℓ 2 \ell_2 ℓ 2 Expected Calibration Error (ECE), considering top-1-to- k k k calibration. result Shows asymptotic normality and different convergence rates for calibrated and miscalibrated models, developing methods to construct valid confidence intervals.
The paper addresses poor calibration in fine-tuned LLMs after preference alignment.
problem Poor calibration in fine-tuned Large Language Models (LLMs) after preference alignment.
method Proposes a calibration-aware fine-tuning approach to restore calibration without compromising model performance.
result Demonstrates the effectiveness of the proposed methods through extensive experiments.
Calibration in 16D disproves Federer's product question.
problem Proving Federer's product question in 16D.
method Adapted Cayley calibration in R^16, constructing a specific calibration Φ.
result Product of two orthogonally supported calibrations is not always a calibration.
New calibration bands for various distributions improve testing for auto-calibration.
problem Testing for auto-calibration in finite samples is challenging.
method Construct calibration bands for the exponential dispersion family using finite sample properties.
result Calibration bands allow for various tests for calibration and auto-calibration.
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
Response calibration is the process of inferring how much the measured data depend on the signal one is interested in. It is essential for any quantitative signal estimation on the basis of the data. Here, we investigate self-calibration methods for linear signal measurements and linear dependence of the response on th…
New calibration measure SSCE ensures truthful prediction, unlike existing measures.
problem Ensuring truthful calibration measures in sequential prediction.
method Introduced a new calibration measure, Subsampled Smooth Calibration Error (SSCE).
result SSCE ensures truthful prediction, while existing measures are far from truthful.
THERMOMETER calibrates LLMs efficiently for diverse tasks.
problem Calibrating large language models is challenging due to computational and versatility issues.
method THERMOMETER learns an auxiliary model for calibrating a LLM using data from multiple tasks.
result THERMOMETER produces better-calibrated responses for new tasks.
Paper detects duality obstruction in smooth calibrations.
problem Detecting duality obstruction in smooth calibrations.
method Examine Lawlor cones and calibrations, use gluing results.
result Existence of Lawlor cones without smooth calibrations.
T-Cal tests model calibration with a minimax optimal test.
problem Detecting mis-calibration of predictive models using a finite validation dataset.
method T-Cal is a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ 2 \ell_2 ℓ 2 -Expected Calibration Error (ECE). result T-Cal is a practical tool for testing the calibration of probabilistic classification methods.
Optimizes calibration error estimators for better classifier trustworthiness.
problem Lack of guidance on selecting and tuning calibration error estimators.
method Reformulates calibration estimation as a regression problem with i.i.d. input pairs.
result Demonstrates the effectiveness of optimized calibration estimators on image classification tasks.
This paper examines how to calibrate ensemble members for better prediction accuracy.
problem Improper calibration of deep neural networks leads to unreliable probability estimates.
method Theoretical analysis and empirical evaluation on CIFAR-100 dataset.
result Well-calibrated ensemble members do not guarantee a well-calibrated ensemble prediction, but a well-calibrated ensemble prediction cannot exceed the average performance of its members.
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.
Proposes a new calibration error estimator for deep neural networks.
problem Improves calibration of deep neural networks, especially for canonical calibration.
method Uses a Dirichlet kernel density estimate to create a low-bias, trainable calibration error estimator.
result Asymptotically converges to true L p L_p L p calibration error, enabling efficient estimation and mini-batch updates. LLMs show surprising confidence in their answers, beyond just tokens.
problem LLMs lack meaningful confidence estimates for their responses.
method Semantic calibration test based on local loss optimality and equivalence classes.
result Base LLMs are semantically calibrated across tasks, contrary to expectations.
New method calibrates models under covariate shifts.
problem Calibration of models can be lost under covariate shifts.
method Importance sampling based approach.
result Efficacy demonstrated on real-world and synthetic datasets.
New method improves calibration in multi-output probabilistic models.
problem Challenges in achieving multivariate calibration in multi-output regression.
method General regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions.
result Significant improvement in calibration across all pre-rank functions without sacrificing predictive accuracy.
SVI and GP surrogates improve calibration of ABMs in epidemiology.
problem Calibrating stochastic ABMs in epidemiology is computationally expensive.
method Stein Variational Inference (SVI) with Gaussian process (GP) surrogates.
result SVI maintains comparable predictive accuracy and calibration effectiveness to MCMC.
Self-calibrating conformal prediction improves interval efficiency and offers a practical alternative.
problem Improving the reliability and uncertainty quantification of machine learning predictions.
method Combines Venn-Abers calibration and conformal prediction for binary and regression problems.
result Improves interval efficiency through model calibration and offers practical alternatives.
KCal calibrates deep networks by embedding logits in a metric space.
problem Overconfident predictions from DNNs, especially in high-risk applications.
method KCal learns a metric space on the penultimate-layer latent embedding and generates predictions using kernel density estimates.
result KCal provides a provable full calibration guarantee and consistently outperforms baselines.
The paper studies the distance from calibration in sequential prediction, proving upper and lower bounds.
problem The challenge is to measure and minimize the deviation from perfect calibration in sequential binary prediction.
method The approach involves proving an O ( T ) O(\sqrt{T}) O ( T ) upper bound and an Ω ( T 1 / 3 ) Ω(T^{1/3}) Ω ( T 1/3 ) lower bound, using structural results and minimax arguments. result An O ( T ) O(\sqrt{T}) O ( T ) upper bound on the calibration distance is achieved, with an Ω ( T 1 / 3 ) Ω(T^{1/3}) Ω ( T 1/3 ) lower bound showing the inherent difficulty.