Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
The paper extends Liouville's theorem to calibrated geometries in various dimensions.
problem Extending Liouville's theorem to calibrated geometries in different dimensions.
method Analyzing Sobolev mappings and calibrations in calibrated geometries.
result Calibrations in certain dimensions have the Liouville property.
New method calibrates uncertainty in molecular property predictions.
problem Uncalibrated uncertainty estimates in molecular property predictions.
method Message Passing Neural Networks with calibrated probabilistic predictive distribution.
result Accurate molecular formation energy predictions with well-calibrated uncertainty.
Time series foundation models are well-calibrated, improving over baseline models.
problem Calibration of time series foundation models for practical applications.
method Systematic evaluations of five time series foundation models and two baselines, assessing calibration, prediction heads, and long-term forecasting.
result Time series foundation models are consistently better calibrated than baseline models and do not show over- or under-confidence.
New algorithm tests model calibration in nearly-linear time.
problem Testing model calibration from samples efficiently.
method Reformulated as minimum-cost flow, solved with dynamic programming.
result Optimal testing problem solved in nearly-linear time.
The paper highlights how machine learning calibrations can be biased by training data.
problem Machine learning calibrations can be biased by the training data, affecting downstream analyses.
method The paper examines simulation-based and data-based calibrations, highlighting their prior dependence and proposing solutions.
result A recently proposed Gaussian Ansatz approach can avoid some biases in simulation-based calibrations.
Estimates proper calibration errors and refinement terms in probabilistic predictions.
problem Lack of a general estimator for proper calibration errors and refinement terms with known statistical properties.
method Proposes a method for consistent, asymptotically unbiased estimation of proper calibration errors and refinement terms.
result Proves the relation between refinement and f-divergences, implying information monotonicity in neural networks.
FlowMO uses Gaussian Processes for molecular property prediction with uncertainty.
problem Predicting molecular properties with uncertainty for small datasets.
method Gaussian Processes implemented in FlowMO, built on GPflow and RDKit.
result Comparable predictive performance to deep learning but superior uncertainty calibration.
Study calibrates high-dimensional binary classifiers using angle between estimator and true weights.
problem Calibrating high-dimensional binary classifiers with provable properties.
method Interpolates with a chance classifier to construct well-calibrated predictor based on angle between estimator and true weights.
result Angular calibration approach is provably well-calibrated in high dimensions, minimizing Bregman divergence.
Improves robustness of propensity score estimators in challenging settings.
problem Limited overlap, small sample sizes, or unbalanced data.
method Extends calibration techniques for propensity score models, focusing on sample-splitting schemes.
result Calibration reduces variance and bias in inverse probability weighting and double/debiased machine learning frameworks.
New calibration measure SCDL improves trust in AI predictions.
problem Improving trust in AI predictions by ensuring they are both actionable and testable.
method Introducing SCDL, a new calibration measure that is fully actionable and testable.
result SCDL is the first calibration measure that is fully actionable and testable.
The paper connects hyperbolicity in calibrated geometry to properties of Smith immersions.
problem Hyperbolicity in calibrated manifolds and its relation to Smith immersions.
method Establishes a theorem relating hyperbolicity to the equicontinuity of Smith immersions, proving a new Schwarz lemma.
result Calibrated hyperbolicity of compact φ φ φ -replete manifolds is equivalent to the equicontinuity of Smith immersions. Certified calibration methods protect model confidence from adversarial attacks.
problem Adversarial attacks degrade model calibration, reducing confidence in predictions.
method Developed certified calibration methods to provide worst-case bounds on calibration under adversarial perturbations.
result Certified calibration methods produce analytic and approximate bounds for the Brier score and expected calibration error.
Histogram binning method proven with guarantees without splitting data.
problem Proving theoretical guarantees for histogram binning without sample splitting.
method Using Markov property of order statistics to prove calibration guarantees for original method.
result Proves histogram binning has strong calibration guarantees without sample splitting.
This work addresses building fair and calibrated models.
problem Building models that are both fair and calibrated.
method Developed a new definition of fairness and showed that group-wise calibration results in fairness. Proposed post-processing techniques and modifications of calibration losses.
result Demonstrated that ensuring group-wise calibration results in a fair model under the new definition of fairness.
Improves model calibration for deep neural networks using proper scores.
problem Calibration errors in deep neural networks are often biased and inconsistent.
method Introduces proper calibration errors related to proper scores.
result Demonstrates the superiority of proper scores over common estimators.
New method improves calibration of BayesCG for better uncertainty quantification.
problem Bayesian conjugate gradient method's poor calibration limits its utility.
method Randomized postiteration strategy to enhance posterior calibration.
result The method improves the distribution of posterior errors and enhances uncertainty quantification.
Proposes a new method for multi-class classification with well-calibrated predictions.
problem Improving the accuracy and reliability of multi-class classification models.
method Trains data in a latent space induced by an ( n − 1 ) (n-1) ( n − 1 ) -dimensional simplex, then extends and fits a regression model. result Demonstrates a well-calibrated classifier with improved prediction and calibration properties.
New calibration measure SSCE ensures truthful prediction, unlike existing measures.
problem Ensuring truthful calibration measures in sequential prediction.
method Introduced a new calibration measure, Subsampled Smooth Calibration Error (SSCE).
result SSCE ensures truthful prediction, while existing measures are far from truthful.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
Proposes isotonic recalibration for insurance pricing to ensure auto-calibration under low signal-to-noise ratio.
problem Ensuring auto-calibration in insurance pricing models to prevent cross-financing.
method Applies isotonic recalibration to regression models to achieve auto-calibration.
result Isotonically recalibrated regression functions have low complexity under low signal-to-noise ratio.
New truthful calibration errors improve model ranking in multiclass prediction.
problem Non-truthful calibration errors can mislead model comparisons.
method Introduced perfectly truthful calibration errors for multiclass predictions.
result Truthful calibration errors preserve decision-theoretic dominance and stabilize model rankings.
New conditions for calibrated submanifolds in Riemannian geometry.
problem Characterizing calibrated submanifolds with extrinsic geometry.
method Introducing compliancy condition and analyzing extrinsic geometry.
result Conditions for extrinsic geometry of calibrated submanifolds.
New study on neural network calibration, linking it to generalization gap.
problem Neural networks lack strong guarantees on calibration.
method Decomposed calibration error into train set and generalization gap.
result Models with small generalization gap are well-calibrated.
New calibration energy measures deviation from calibrated geometry, enabling mean curvature flow in infinite volumes.
problem Mean curvature flow in infinite volumes with finite energy.
method Introducing calibration energy and proving its dissipation identity for mean curvature flows.
result Every proper self-expander with finite calibration energy is a plane in all dimensions and codimensions.
We introduce and study the notion of plurisubharmonic functions in calibrated geometry. These functions generalize the classical plurisubharmonic functions from complex geometry and enjoy their important properties. Moreover, they exist in abundance whereas the corresponding pluriharmonics are generally quite scarce. A…
Scaffolding sets improve predictor correctness across subsets.
problem Ensuring predictor correctness across multiple subsets.
method Inspired by neural nets, constructing scaffolding sets to ensure correctness.
result Scaffolding sets ensure predictor correctness, not just calibration.
Isotonic regression binning affects calibration statistics of machine learning models.
problem Isotonic regression binning introduces aleatoric uncertainty in calibration statistics.
method Calibration error statistics are recalibrated using isotonic regression, which produces stratified uncertainties.
result Stratified uncertainties lead to significant differences in bin-based calibration statistics.
The paper proves geodesics and conic sections are length-minimizing under specific metrics.
problem Finding shortest paths in complex geometries.
method Calibrations and conformal metrics.
result Geodesics and conic sections are length-minimizing.
The paper studies calibration in ML models for wireless networks, showing key theoretical and practical insights.
problem Ensuring ML models in wireless networks deliver well-calibrated confidence scores for reliable decision-making.
method Theoretical analysis and simulation-based experiments using Platt scaling and isotonic regression.
result Well-calibrated models improve the system's minimum achievable OP and are part of a broader class of predictors.
In this paper we introduce and study the notion of plurisubharmonic functions in calibrated geometry. These functions generalize the classical plurisubharmonic functions from complex geometry and enjoy many of their important properties. Moreover, they exist in abundance whereas the corresponding pluriharmonics are gen…
The paper studies how to extend local calibration pairs to global ones in various situations. As a result, new discoveries involving mass-minimizing properties are exhibited. In particular, we show that a R \mathbb R R -homologically nontrivial connected submanifold M M M of a smooth Riemannian manifold X X X is homologically…
The paper reviews and extends calibration concepts for classification and regression.
problem Formalizing compatibility between probabilistic predictions and outcomes.
method Review and extension of existing calibration concepts, introduction of new concepts.
result Hierarchical relations between calibration concepts for various data types.
Paper introduces a method to improve survival model calibration without sacrificing discrimination.
problem Survival models struggle to balance accurate ranking and event prediction.
method Uses conformal regression to enhance calibration without compromising discrimination.
result The approach improves model calibration across 11 real-world datasets.
Reassesses calibration metrics in machine learning models.
problem Inconsistent reporting of calibration metrics in recent literature.
method Calibration-based decomposition of Bregman divergences, visualization of calibration and generalization error.
result New visualization technique for detecting trade-offs between calibration and generalization.
New test assesses probabilistic model calibration without expensive approximations.
problem Assessing calibration of probabilistic models with scores.
method Kernel Calibration Conditional Stein Discrepancy (KCCSD) test using new score-based kernels.
result Control over type-I error with improved scalability and efficiency.
Optimizing proper loss yields calibrated models under specific conditions.
problem Understanding when optimizing proper loss functions leads to calibrated predictions.
method Local optimality condition and Lipschitz functions.
result Predictors with local optimality are nearly calibrated and nearly locally optimal.
Deep ensembles don't necessarily improve calibration in low data regimes.
problem Calibration issues in deep learning models, especially in low data regimes.
method Examination of data-augmentation, ensembling, and post-processing calibration methods.
result Standard ensembling techniques can lead to less calibrated models in low data regimes.
The paper addresses decision making with partially calibrated forecasts, offering a robust approach.
problem Developing a decision-making strategy for forecasts that are only partially calibrated.
method A minimax approach to mapping predictions to actions, considering worst-case distributions.
result The minimax optimal decision rule is to trust predictions and act accordingly, even for partially calibrated forecasts.
The paper explores various forms of calibration scores and their implications for fairness.
problem The evaluation of probabilistic predictions through calibration.
method The authors organize three grouping choices and one agglomeration of group errors, providing a framework for comparing and creating new calibration scores.
result The study demonstrates that appropriate choices of grouping can provide notions of (sub-)group or individual fairness.
The paper studies the distance from calibration in sequential prediction, proving upper and lower bounds.
problem The challenge is to measure and minimize the deviation from perfect calibration in sequential binary prediction.
method The approach involves proving an O ( T ) O(\sqrt{T}) O ( T ) upper bound and an Ω ( T 1 / 3 ) Ω(T^{1/3}) Ω ( T 1/3 ) lower bound, using structural results and minimax arguments. result An O ( T ) O(\sqrt{T}) O ( T ) upper bound on the calibration distance is achieved, with an Ω ( T 1 / 3 ) Ω(T^{1/3}) Ω ( T 1/3 ) lower bound showing the inherent difficulty. TCE measures calibration error with a test-based approach.
problem Measuring calibration error of probabilistic binary classifiers.
method TCE uses a novel loss function based on a statistical test.
result TCE offers clear interpretation, consistent scale, and enhanced visual representation.
We equip many non compact non simply connected surfaces with smooth Riemannian metrics whose isoperimetric profile is smooth, a highly non generic property. The computation of the profile is based on a calibration argument, a rearrangement argument, the Bol-Fiala curvature dependent inequality, together with new result…
LiST improves neural network robustness and calibration without manual tuning.
problem Developing robust and calibrated neural networks simultaneously.
method Lipschitz Scaling Training (LiST) that iteratively adjusts the global Lipschitz constant.
result LiST yields an out-of-the-box calibrated network with competitive accuracy and robustness.
Two methods improve calibration of probabilistic classifiers, especially for multi-class problems.
problem Improving calibration of probabilistic classifiers, especially for multi-class problems.
method Two techniques: reduced calibration and class-wise calibration.
result Class-wise reduced calibration algorithms reduce prediction and per-class calibration errors.
New tan-concavity property for Lagrangian phase operators helps in studying dHYM metrics.
problem Lack of concavity in Lagrangian phase operator for dHYM metrics.
method Introduce tangent Lagrangian phase flow (TLPF) on almost calibrated (1,1)-forms.
result TLPF exists for all positive time and converges to dHYM metrics under certain conditions.
Proposes isotonic regression for calibrating Deep Cox models' survival probabilities.
problem Poor calibration of Deep Cox models' survival probabilities.
method Isotonic regression for post hoc calibration of Deep Cox models.
result Establishes favorable theoretical guarantees and demonstrates empirical effectiveness.
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems tha…