arXiv research

A new perfectly truthful calibration measure improves prediction reliability.

New calibration measure SSCE ensures truthful prediction, unlike existing measures.

This post introduces model calibration and evaluation measures, highlighting issues with a common measure.

New calibration measure SCDL improves trust in AI predictions.

In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class…

Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, ha…

Response calibration is the process of inferring how much the measured data depend on the signal one is interested in. It is essential for any quantitative signal estimation on the basis of the data. Here, we investigate self-calibration methods for linear signal measurements and linear dependence of the response on th…

New truthful calibration errors improve model ranking in multiclass prediction.

Bayesian framework predicts aerodynamic uncertainty from sparse measurements.

Survey on calibration in machine learning, viewing it as indistinguishability.

The paper introduces a spline-based method for calibrating neural networks.

Study explores calibration properties in neural architectures.

For many applications it is critical to know the uncertainty of a neural network's predictions. While a variety of neural network parameter estimation methods have been proposed for uncertainty estimation, they have not been rigorously compared across uncertainty measures. We assess four of these parameter estimation m…

TCE measures calibration error with a test-based approach.

The paper studies the distance from calibration in sequential prediction, proving upper and lower bounds.

New method calibrates classifier probabilities with guaranteed coverage.

New algorithm tests model calibration in nearly-linear time.

Novel framework for unbiased confidence estimates in object detection.

Bayesian calibration for BCP self-assembly models using image data and measure transport.

New method calibrates asynchronous, error-prone covariates for longitudinal data.

Paper proposes a unified framework for evaluating calibration of probabilistic models.

Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popu…

New framework assesses LM uncertainty without thresholding.

Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliabil…

Survey of methods to calibrate neural network predictions.

New framework for evaluating multiclass classifier calibration.

A new calibration metric bridges testability and actionability.

Certified calibration methods protect model confidence from adversarial attacks.

Simple algorithm achieves distance to calibration error of at most 2√T+1.

New method improves decision-making accuracy without complex calculations.

There is resurging interest, in statistics and machine learning, in solvers for ordinary differential equations (ODEs) that return probability measures instead of point estimates. Recently, Conrad et al. introduced a sampling-based class of methods that are 'well-calibrated' in a specific sense. But the computational c…

Smooth calibration improves forecast reliability even with leaked information.

This paper calibrates distribution models from PELVE values.

Improves confidence calibration in neural networks by smoothing labels based on class similarity.

Develops confidence intervals for ECE, a measure of model calibration.

This paper addresses measurement errors in high-dimensional compositional data using a log-contrast model calibration approach.

This paper improves confidence measurement in deep metric learning models.

The paper proposes a method to produce well-calibrated predictions in regression tasks using maximum mean discrepancy.

Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to …

A new method calibrates forecasts without sacrificing expertise.

New algorithms adaptively calibrate predictions in non-stationary environments, matching optimal rates.

New methods reduce bias in estimating calibration error.

The calibration of a measurement device is crucial for every scientific experiment, where a signal has to be inferred from data. We present CURE, the calibration uncertainty renormalized estimator, to reconstruct a signal and simultaneously the instrument's calibration from the same data without knowing the exact calib…

Proposes a method to calibrate data for more accurate linear correlation testing.

New calibration energy measures deviation from calibrated geometry, enabling mean curvature flow in infinite volumes.

A new method for measuring prediction uncertainty in classifiers.

Proposes an accuracy-preserving calibration method for DNNs.

KCal calibrates deep networks by embedding logits in a metric space.