New methods for better uncertainty prediction in ML.
problem Insufficient calibration in machine learning regression.
method Conditional calibration with respect to input features (adaptivity).
result Consistency and adaptivity are complementary, and good consistency does not guarantee good adaptivity.
Calibrates network confidence for unsupervised domain adaptation.
problem Calibrating a model trained on a source domain to a target domain without labeled data.
method Estimates network accuracy on the target domain and calibrates prediction confidence directly in the target domain.
result Significantly outperforms existing methods across standard datasets.
Conformal Bayes under label shift: post-hoc calibration vs. in-training adaptation
problem Bayesian prediction sets under label shift
method Post-hoc calibration vs. In-training adaptation
result Both strategies achieve valid coverage equally in an unbiased training regime
Improves model calibration and selection in unsupervised domain adaptation.
problem Distribution shifts in unsupervised domain adaptation.
method Developed a novel importance weighted group accuracy estimator.
result Improves state-of-the-art performances by 22% in model calibration and 14% in model selection.
New method calibrates models under covariate shifts.
problem Calibration of models can be lost under covariate shifts.
method Importance sampling based approach.
result Efficacy demonstrated on real-world and synthetic datasets.
Adaptive calibration improves model accuracy with fewer simulations.
problem Inefficient calibration of complex models using fixed designs.
method Bayesian adaptive experimental design to optimize simulation runs.
result The method achieves better parameter estimation with fewer simulations.
New algorithms adaptively calibrate predictions in non-stationary environments, matching optimal rates.
problem Designing online prediction algorithms that adapt to varying levels of non-stationarity.
method Epoch-based scheduling and non-uniform partitioning of the prediction space.
result Achieves adaptive calibration guarantees under multiple measures with optimal rates.
TransCal calibrates DA models with lower bias and variance.
problem Calibrating DA models to estimate accurate predictive uncertainty.
method Transferable Calibration (TransCal) in a unified hyperparameter-free optimization framework.
result TransCal achieves more accurate calibration with lower bias and variance.
Calibrated probabilistic solvers improve accuracy of ODE estimates.
problem Uncertainty in probabilistic ODE solutions is not well-calibrated for adaptive step sizes.
method Introduce and assess several calibration methods for probabilistic ODE solvers.
result Calibration methods interact efficiently with adaptive step-size selection, improving posteriors.
Algorithm calibrates predictions for covariate shift using domain adaptation.
problem Uncertainty estimates overestimate certainty when real-world data differs from training data.
method Uses importance weighting and learns a feature map to equalize distributions.
result Outperforms existing approaches in calibrated prediction when covariate shift occurs.
CAT improves domain adaptation for fault diagnosis by calibrating teacher network predictions.
problem Performance drops in deep learning models when applied to different data distributions.
method CAT uses post-hoc calibration techniques to calibrate predictions of the teacher network during self-training.
result CAT achieves state-of-the-art performance on most transfer tasks in domain-adaptive IFD.
AdapTable adapts tabular models to shifts without source data, improving HELOC performance.
problem Distribution shifts in tabular data threaten model performance.
method Shift-aware uncertainty calibrator and label distribution handler.
result Up to 16% improvement on HELOC dataset.
Adaptive uncertainty quantification improves black-box model predictions in generative AI.
problem Improving uncertainty quantification for black-box models in generative AI.
method Adaptive partitioning and local calibration of conformity scores.
result Local tightening of uncertainty sets with adaptive bands.
This paper tackles over-certainty in test-time adaptation models, proposing a solution to improve calibration.
problem Over-certainty in predictions caused by domain shifts, leading to misplaced trust.
method Introduces a certainty regularizer that dynamically adjusts pseudo-label confidence based on backbone entropy and logit norm.
result Achieves state-of-the-art performance in terms of Expected Calibration Error and Negative Log Likelihood, while maintaining accuracy.
New method calibrates classifier probabilities with guaranteed coverage.
problem Inaccurate probability estimates by classifiers in high-risk applications.
method Adaptive temperature scaling algorithm for conformal prediction.
result Improves calibration error measures and standard metrics across various tasks.
Researchers calibrate an adaptive Farmer-Joshi model to recover stylized facts in financial markets.
problem Recovering stylized facts in financial markets using the Farmer-Joshi model.
method Calibrated an adaptive Farmer-Joshi model using genetic and Nelder-Mead algorithms, incorporating agent adaptation.
result The adaptive model recovers additional stylized facts, including auto-correlations and kurtosis, compared to the original model.
Calibrated PRMs improve inference efficiency for LLMs by dynamically adjusting compute budgets.
problem Poor calibration of PRMs leads to overestimation of success probabilities in partial reasoning steps.
method Quantile regression for calibration, instance-adaptive scaling (IAS) framework.
result Calibrated PRMs reduce inference costs while maintaining accuracy, especially on confident problems.
Unsupervised domain adaptation (UDA) aims at inferring class labels for unlabeled target domain given a related labeled source dataset. Intuitively, a model trained on source domain normally produces higher uncertainties for unseen data. In this work, we build on this assumption and propose to adapt from source to targ…
Corrects bias in LLM-as-a-judge evaluations using adaptive calibration.
problem Bias in LLM evaluations due to imperfect sensitivity and specificity.
method Plug-in framework with confidence intervals accounting for test and calibration dataset uncertainties.
result LML-based evaluation yields more reliable estimates than human-only evaluation.
This paper rethinks confidence calibration under covariate shifts.
problem Calibration methods struggle with covariate shifts and unstable importance weighting.
method Derives Expectation consistency condition and proposes Expectation consistency loss (ECL).
result ECL loss is compatible with various types of calibration and has the same sample complexity as ECE.
New federated learning methods improve model performance on non-IID data.
problem Improving model performance on non-IID decentralized data.
method Proposed Federated AGMs using adaptive gradient methods with first-order and second-order momenta.
result The proposed Federated AGMs converge to a first-order stationary point under non-IID and unbalanced data settings for nonconvex optimization.
Improved segmentation model adaptation for new domains.
problem Reduced performance of pre-trained models on new domains.
method Calculated soft-label prototypes and predicted closest to class probabilities.
result Significant performance improvements on synthetic-to-real segmentation.
ELSA efficiently adapts to label shift without post-prediction calibrations.
problem Domain adaptation with label shift across training and testing datasets.
method Moment-matching framework based on influence function geometry; solves linear systems for adaptation weights.
result ELSA estimator is n \sqrt{n} n -consistent and asymptotically normal, achieving state-of-the-art estimation performance. Study aggregation of statistical evidence under unknown dependence using group-invariance.
problem Aggregating statistical evidence under unknown and complex dependence structures.
method Develops a framework using group-invariance and permutation-based constructions to aggregate evidence across transformed datasets.
result Shows uniform improvement in critical values for single-batch aggregation over deterministic calibrations, adapting to unknown dependence structures.
T-Cal tests model calibration with a minimax optimal test.
problem Detecting mis-calibration of predictive models using a finite validation dataset.
method T-Cal is a minimax optimal test for calibration based on a debiased plug-in estimator of the ℓ 2 \ell_2 ℓ 2 -Expected Calibration Error (ECE). result T-Cal is a practical tool for testing the calibration of probabilistic classification methods.
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems tha…
Improves model calibration by addressing adversarially unrobust inputs.
problem Miscalibrated predictions and lack of adversarial robustness in neural networks.
method AR-AdaLS, an adaptive label smoothing method that considers adversarial robustness.
result AR-AdaLS improves model calibration, even under distributional shifts.
MOPI optimizes flexible set-valued mappings to achieve superior shape adaptivity in conformal prediction.
problem Challenges in achieving valid conditional coverage in conformal prediction.
method Minimax Optimization Predictive Inference (MOPI) framework that optimizes over a flexible class of set-valued mappings.
result MOPI achieves superior shape adaptivity and maintains a principled connection to mean squared coverage error.
New bin-wise scaling methods improve prediction uncertainty calibration for machine learning.
problem Improving prediction uncertainty calibration for machine learning regression.
method Adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and feature-based binning.
result Improved adaptivity and consistency in prediction uncertainty calibration.
AECF improves multimodal inference robustness and calibration.
problem Robustness and calibration issues in multimodal systems with missing inputs.
method Adaptive Entropy-Gated Contrastive Fusion (AECF) layer.
result Improves masked-input mAP by +18 pp at a 50% drop rate.
Improved IVON boosts LoRA model accuracy and calibration.
problem Improving the accuracy and calibration of large language models.
method Replaced AdamW with IVON for finetuning Llama-2.
result IVON improves Llama-2 accuracy by 2.8% and calibration error by 4.6%.
Calibration in 16D disproves Federer's product question.
problem Proving Federer's product question in 16D.
method Adapted Cayley calibration in R^16, constructing a specific calibration Φ.
result Product of two orthogonally supported calibrations is not always a calibration.
Label shift refers to the phenomenon where the prior class probability p(y) changes between the training and test distributions, while the conditional probability p(x|y) stays fixed. Label shift arises in settings like medical diagnosis, where a classifier trained to predict disease given symptoms must be adapted to sc…
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, ha…
Improves forecast calibration for extreme events using modified loss functions.
problem Improperly specified models do not issue calibrated forecasts for extreme events.
method Adapting loss functions based on weighted scoring rules and tail miscalibration regularization.
result Calibrated forecasts for extreme wind speeds can be improved by suitable adaptations to the loss function during model training.
Spectral Adaptive Conformal Prediction for Structured Non-Exchangeable Data
problem Improving prediction intervals for non-exchangeable time-indexed datasets
method Spectral adaptive conformal prediction
result Improves on fixed spectral weighting while monitoring uncertainty changes
Adaptive method improves prediction intervals with global coverage guarantees and local error distribution.
problem Global coverage guarantees of conformal regression are often violated by local error distributions.
method Adaptive Conformal Regression with Jackknife+ Rescaled Scores
result Improves local coverage without sacrificing global coverage, especially in low-data regimes.
Two approaches improve conformal Bayes for label shift, one post-hoc and one in-training.
problem Improving prediction sets for target domain under label shift.
method Two complementary approaches: post-hoc calibration and in-training adaptation.
result In-training adaptation achieves up to 43% width reduction at unchanged coverage.
TS improves class coverage but reduces CP set size, offering a trade-off.
problem Combining temperature scaling with conformal prediction for deep classifiers.
method Empirical study and mathematical theory of TS's effect on CP.
result TS allows trading prediction set size and conditional coverage.
TCP provides well-calibrated prediction intervals for nonstationary time series.
problem Nonstationary time series forecasting with well-calibrated prediction intervals.
method Temporal Conformal Prediction (TCP) couples a modern quantile forecaster with a rolling split-conformal calibration layer.
result TCP achieves near-nominal coverage, providing slightly wider intervals than Historical Simulation.
The paper addresses probability calibration for incomplete sequences.
problem Improving probability estimates from incomplete sequences.
method Adapting traditional calibration techniques to sequences of varying lengths.
result Proposed methods improve probability calibration for modern sequential models.
GP model calibration improves optimization algorithm performance.
problem GP model uncertainty calibration issues degrade optimization performance.
method Kernel validation procedure to calibrate GP predictions.
result Proper calibration enhances optimization algorithm convergence.
In many classification problems it is desirable to output well-calibrated probabilities on the different classes. We propose a robust, non-parametric method of calibrating probabilities called SplineCalib that utilizes smoothing splines to determine a calibration function. We demonstrate how applying certain transforma…
New method speeds up NIR spectroscopy calibration by 400x.
problem Efficient preprocessing selection in NIR spectroscopy.
method Operator-adaptive PLS and Ridge regression.
result Significant reduction in fitting time with comparable prediction quality.
LCMQR improves prediction intervals by adapting to local heteroscedasticity.
problem Efficient and adaptive prediction intervals for local heteroscedasticity.
method LCMQR combines multi-quantile information with kernel-based localization.
result LCMQR constructs tighter intervals than prior methods, especially in heterogeneous environments.
LoBoost improves local conformal prediction for gradient-boosted trees without extra data splits.
problem Quantifying uncertainty in gradient-boosted tree predictions.
method Model-native local conformal prediction using leaf structure.
result Competitive interval quality and improved test MSE with large calibration speedups.
Extends DML for parametric problems, improving accuracy and efficiency in pricing and calibration.
problem Improving precision and efficiency in pricing and calibration for parametric problems.
method Exploits derivative information, uses adaptive parameter sampling, constructs pricing surrogates, and optimizes globally.
result Demonstrates improved accuracy and efficiency in pricing and calibration for complex models.
Cross-regularization adapts model complexity during training.
problem Manual tuning of model complexity for overfitting prevention.
method Directly adapts regularization parameters through validation gradients during training.
result Organic emergence of architecture-specific regularization during training.