New method learns SIMs with arbitrary monotone activations without strong distributional assumptions.
problem Learning Single-Index Models with arbitrary monotone activations.
method Based on omniprediction with calibrated multiaccuracy and Bregman divergences.
result First agnostic learning result for SIMs with arbitrary monotone activations.
The paper shows how to use proxy attributes for fairness in machine learning models with missing sensitive group data.
problem Measuring and enforcing fairness in machine learning models with incomplete sensitive group data.
method Using proxy-sensitive attributes to derive upper bounds on multiaccuracy and multicalibration violations and adjust models to satisfy these fairness notions.
result Provable upper bounds on multiaccuracy and multicalibration violations can be derived using proxy-sensitive attributes in the absence of sensitive group data.
Prediction systems are successfully deployed in applications ranging from disease diagnosis, to predicting credit worthiness, to image recognition. Even when the overall accuracy is high, these systems may exhibit systematic biases that harm specific subpopulations; such biases may arise inadvertently due to underrepre…
Improves multi-objective learning by adapting to local subintervals.
problem Learning a predictor satisfying multiple objectives in an online, changing data setting.
method Adapting an existing multi-objective learning method with an adaptive online algorithm.
result Improves predictions over subgroups and remains robust under distribution shift.
Unified theory and practical insights for multicalibration boosting.
problem Ensuring fairness and robustness in predictions over diverse functions.
method Developed a unified framework for multicalibration boosting under weaker conditions.
result Boosting iterates converge to a Bregman projection, characterizing multicalibration.
Proposes new method for calibrating treatment effect predictors.
problem Calibrating predictors of heterogeneous treatment effects.
method Causal isotonic calibration and cross-calibration.
result Achieves fast calibration rates under weak conditions.
Proposes top-label calibration and M2B framework for multiclass to binary calibration.
problem Multiclass calibration and interpretation issues.
method Top-label calibration and M2B reduction framework.
result M2B + HB achieves lower calibration error than other methods.
Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
A new perfectly truthful calibration measure improves prediction reliability.
problem Improving the reliability of predictions by ensuring they are conditionally unbiased.
method Designing a simple, perfectly truthful calibration measure called ATB.
result ATB is the first perfectly truthful calibration measure in the batch setting.
New truthful calibration errors improve model ranking in multiclass prediction.
problem Non-truthful calibration errors can mislead model comparisons.
method Introduced perfectly truthful calibration errors for multiclass predictions.
result Truthful calibration errors preserve decision-theoretic dominance and stabilize model rankings.
New framework for evaluating multiclass classifier calibration.
problem Ensuring classifiers are well-calibrated for trustworthy predictions.
method Utility Calibration framework that measures calibration error relative to a utility function.
result Unified and robust interpretation of existing calibration metrics.
We propose a new framework to improve the calibration of neural networks.
problem Improving the accuracy of model confidence predictions.
method Introducing a differentiable surrogate for expected calibration error (DECE) and a meta-learning framework to optimise model hyper-parameters for validation set calibration.
result Achieved competitive performance with existing calibration approaches.
Certified calibration methods protect model confidence from adversarial attacks.
problem Adversarial attacks degrade model calibration, reducing confidence in predictions.
method Developed certified calibration methods to provide worst-case bounds on calibration under adversarial perturbations.
result Certified calibration methods produce analytic and approximate bounds for the Brier score and expected calibration error.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
Post-processing predictors reduces calibration errors for decision-making.
problem Predictors with low calibration error for machine learning may have high error for decision-making.
method Post-processing with ε distance to calibration adds noise to make predictions differentially private.
result Post-processing achieves O(√ε) ECE and CDL, asymptotically optimal.
A new calibration metric bridges testability and actionability.
problem Combining testability and actionable insights for forecast probabilities.
method Cutoff Calibration Error (CCE) that assesses calibration over intervals of forecasted probabilities.
result Cutoff Calibration Error is both testable and actionable.
New study on neural network calibration, linking it to generalization gap.
problem Neural networks lack strong guarantees on calibration.
method Decomposed calibration error into train set and generalization gap.
result Models with small generalization gap are well-calibrated.
A new method for multiclass calibration using vector quantization.
problem Challenges in multiclass calibration, especially in high-stakes settings.
method Compositional approach via Vector Quantization (VQ) to learn region-specific calibration maps.
result Significant improvements in local calibration with competitive global calibration and predictive performance.
This post introduces model calibration and evaluation measures, highlighting issues with a common measure.
problem Ensuring model confidence accurately reflects true outcomes.
method Explains common calibration definition, ECE, and its drawbacks.
result New evaluation measures needed for comprehensive model calibration.
Unified calibration metrics improve forecast sharpness and accuracy.
problem Improving the sharpness of probabilistic forecasts while maintaining calibration.
method Kernel-based calibration metrics that unify and generalize existing methods for classification and regression.
result Enhanced calibration, sharpness, and decision-making across various tasks.
New decision-theoretic calibration error metric improves prediction reliability.
problem Improving the reliability of predictions for decision-making.
method Proposed Calibration Decision Loss (CDL) and an efficient algorithm to achieve near-optimal CDL.
result Near-optimal CDL guarantees vanishing payoff loss from miscalibration.
Cone structures over minimal products can't be calibrated smoothly.
problem Calibrating cones over minimal products with smooth calibrations.
method Extending a key result from [Zha26], showing obstruction.
result Cone structures over minimal products cannot be calibrated by smooth calibrations.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Smooth calibration improves forecast reliability even with leaked information.
problem Improving forecast reliability with leaked information.
method Combining nearby forecasts to ensure smooth calibration, which can be guaranteed by deterministic procedures.
result Smooth calibration can be guaranteed by deterministic procedures even with leaked forecasts, and it yields uncoupled finite-memory dynamics in games.
New method improves model calibration efficiency and accuracy.
problem Improving model calibration for better probability estimates.
method Scaling-binning calibrator method that reduces variance and ensures calibration.
result 35% lower calibration error than histogram binning and guarantees true calibration.
Planes are the only calibrated submanifolds with flat normal bundles.
problem Characterizing submanifolds with specific geometric properties.
method Using constant-coefficient differential forms and parallel calibrations.
result Calibrated submanifolds with flat normal bundles are planes.
We describe a family of calibrations arising naturally on a hyperkähler manifold M. These calibrations calibrate the holomorphic Lagrangian, holomorphic isotropic and holomorphic coisotropic subvarieties. When M is an HKT (hyperkaehler with torsion) manifold with holonomy SL(n,H), we construct another fam…
Simple proof shows forecasts can be calibrated in a few periods.
problem Ensuring forecasts are calibrated over multiple periods.
method Uses minimax theorem to prove existence and guarantees calibration error.
result Calibration can be achieved in N3 periods with error at most 1/N. Survey of methods to calibrate neural network predictions.
problem Ensuring neural networks provide accurate confidence levels.
method Empirical comparison of calibration methods.
result Various techniques for calibrating neural networks.
This paper rethinks confidence calibration under covariate shifts.
problem Calibration methods struggle with covariate shifts and unstable importance weighting.
method Derives Expectation consistency condition and proposes Expectation consistency loss (ECL).
result ECL loss is compatible with various types of calibration and has the same sample complexity as ECE.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
New conditions for calibrated submanifolds in Riemannian geometry.
problem Characterizing calibrated submanifolds with extrinsic geometry.
method Introducing compliancy condition and analyzing extrinsic geometry.
result Conditions for extrinsic geometry of calibrated submanifolds.
Survey of Optimal Transport for model calibration.
problem Model calibration using Optimal Transport.
method General framework and numerical algorithms for various models.
result Calibration of volatility models and path-dependent options.
Develops confidence intervals for ECE, a measure of model calibration.
problem Ensuring the calibration of probabilistic predictions in machine learning models.
method Develops confidence intervals for the ℓ2 Expected Calibration Error (ECE), considering top-1-to-k calibration. result Shows asymptotic normality and different convergence rates for calibrated and miscalibrated models, developing methods to construct valid confidence intervals.
The paper addresses poor calibration in fine-tuned LLMs after preference alignment.
problem Poor calibration in fine-tuned Large Language Models (LLMs) after preference alignment.
method Proposes a calibration-aware fine-tuning approach to restore calibration without compromising model performance.
result Demonstrates the effectiveness of the proposed methods through extensive experiments.
New calibration measures for multi-class classification improve model accuracy.
problem Calibration of multi-class classification models is insufficient for safety-critical applications.
method Developed new calibration measures and estimators for multi-class classification.
result Proposed estimators improve interpretability and accuracy of calibration measures.
Calibration in 16D disproves Federer's product question.
problem Proving Federer's product question in 16D.
method Adapted Cayley calibration in R^16, constructing a specific calibration Φ.
result Product of two orthogonally supported calibrations is not always a calibration.
New calibration bands for various distributions improve testing for auto-calibration.
problem Testing for auto-calibration in finite samples is challenging.
method Construct calibration bands for the exponential dispersion family using finite sample properties.
result Calibration bands allow for various tests for calibration and auto-calibration.
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
Response calibration is the process of inferring how much the measured data depend on the signal one is interested in. It is essential for any quantitative signal estimation on the basis of the data. Here, we investigate self-calibration methods for linear signal measurements and linear dependence of the response on th…
New calibration measure SSCE ensures truthful prediction, unlike existing measures.
problem Ensuring truthful calibration measures in sequential prediction.
method Introduced a new calibration measure, Subsampled Smooth Calibration Error (SSCE).
result SSCE ensures truthful prediction, while existing measures are far from truthful.
THERMOMETER calibrates LLMs efficiently for diverse tasks.
problem Calibrating large language models is challenging due to computational and versatility issues.
method THERMOMETER learns an auxiliary model for calibrating a LLM using data from multiple tasks.
result THERMOMETER produces better-calibrated responses for new tasks.
Paper detects duality obstruction in smooth calibrations.
problem Detecting duality obstruction in smooth calibrations.
method Examine Lawlor cones and calibrations, use gluing results.
result Existence of Lawlor cones without smooth calibrations.
T-Cal tests model calibration with a minimax optimal test.
problem Detecting mis-calibration of predictive models using a finite validation dataset.
method T-Cal is a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ2-Expected Calibration Error (ECE). result T-Cal is a practical tool for testing the calibration of probabilistic classification methods.
Optimizes calibration error estimators for better classifier trustworthiness.
problem Lack of guidance on selecting and tuning calibration error estimators.
method Reformulates calibration estimation as a regression problem with i.i.d. input pairs.
result Demonstrates the effectiveness of optimized calibration estimators on image classification tasks.
Improves calibration of regression models without requiring additional data.
problem Poor calibration of regression models leading to unreliable predictions.
method Quantile regularizer based on cumulative KL divergence.
result Significantly improves calibration for regression models trained with Dropout VI and Deep Ensembles.
This paper examines how to calibrate ensemble members for better prediction accuracy.
problem Improper calibration of deep neural networks leads to unreliable probability estimates.
method Theoretical analysis and empirical evaluation on CIFAR-100 dataset.
result Well-calibrated ensemble members do not guarantee a well-calibrated ensemble prediction, but a well-calibrated ensemble prediction cannot exceed the average performance of its members.
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.