Paper analyzes ECE bias and provides bounds for its estimation.
problem Understanding the estimation bias in ECE for machine learning models.
method Information-theoretic approach to analyze bias in uniform mass and uniform width binning strategies.
result Established upper bounds on ECE estimation bias and optimal number of bins.
New methods correct bias in LLM-as-a-Judge evaluations, but reliability depends on judge quality and model calibration.
problem Systematic bias in LLM-as-a-Judge evaluations using naive estimators.
method Analytical results, simulations, and real-data case study to diagnose reliability of corrected estimates.
result Corrected estimates, especially shared-calibration comparisons, can be unreliable under certain conditions.
Proposes a three-stage debiasing framework to improve out-of-distribution accuracy.
problem Inaccurate uncertainty estimations in bias-only models damage ensemble-based debiasing performance.
method Calibrates the bias-only model to improve its uncertainty estimations, creating a three-stage ensemble-based debiasing framework.
result The three-stage debiasing framework consistently outperforms traditional methods in out-of-distribution accuracy.
New methods reduce bias in estimating calibration error.
problem Reducing bias in estimating calibration error.
method Synthesizing model outputs and using equal-mass bins.
result Two reliable calibration-error estimators found: debiased estimator and ECE_sweep.
TransCal calibrates DA models with lower bias and variance.
problem Calibrating DA models to estimate accurate predictive uncertainty.
method Transferable Calibration (TransCal) in a unified hyperparameter-free optimization framework.
result TransCal achieves more accurate calibration with lower bias and variance.
BRPC online Bayesian calibration handles gradual and abrupt system changes.
problem Aligning model outputs with field observations in evolving systems.
method Bayesian Recursive Projected Calibration (BRPC) for streaming data under simulator mismatch and nonstationarity.
result Improves calibration accuracy under gradual changes and robustness under abrupt regime shifts.
Corrects bias in LLM-as-a-judge evaluations using adaptive calibration.
problem Bias in LLM evaluations due to imperfect sensitivity and specificity.
method Plug-in framework with confidence intervals accounting for test and calibration dataset uncertainties.
result LML-based evaluation yields more reliable estimates than human-only evaluation.
Deep learning models show bias and variance are aligned, not in trade-off.
problem The classical bias-variance trade-off in deep learning models.
method Empirical evidence and theoretical analysis of bias and variance in deep learning models.
result Squared bias is approximately equal to variance for correctly classified sample points in deep learning models.
Proposes stabilized weights for causal inference using isotonic calibration.
problem Stability and bias issues in inverse propensity weighting.
method Post-hoc isotonic calibration of inverse propensity weights.
result Improves performance of doubly robust estimators of average treatment effect.
Proposes efficient calibration for indoor localization models.
problem Calibration data scarcity in wireless indoor localization.
method Uses synthetic labels and prediction sets to fine-tune a predictor and estimate bias.
result Yields rigorous coverage guarantees for prediction sets.
We rebias estimates to improve interval calibration and prediction accuracy.
problem Constructing accurate intervals for noisy and biased estimates.
method Empirical Bayes rebiasing strategy that learns bias distribution from data.
result Substantial precision gains in prediction-powered inference.
Recommender systems are personalized: we expect the results given to a particular user to reflect that user's preferences. Some researchers have studied the notion of calibration, how well recommendations match users' stated preferences, and bias disparity the extent to which mis-calibration affects different user grou…
Tree-based models biased when trained on imbalanced data, requiring new calibration methods.
problem Bias in tree-based models trained on imbalanced datasets.
method Analytical calibration of random forest models, demonstrating bias in decision trees.
result Calibrating tree-based models on imbalanced data negatively impacts predictions, especially for the minority class.
Randomized predictions ensure fair and accurate individual calibration in machine learning.
problem Systematic bias in typical calibration methods leads to unfair predictions for certain subgroups.
method Randomization of predictions to enforce individual calibration, trading off bias with variance.
result Randomized regression functions are more calibrated for arbitrary subgroups and achieve higher utility.
The study examines when to trust confidence thresholding in pseudo-labelling regression.
problem Calibrated probabilities from classifiers used for pseudo-labelling need careful handling to avoid bias in downstream regression.
method Developed a diagnostic apparatus to predict and bound the bias induced by confidence thresholding, derived a closed-form expression for the attenuation bias.
result The bias can be predicted from the residual score variance V∗, motivating a structural separation between classifier features and downstream controls. The paper highlights how machine learning calibrations can be biased by training data.
problem Machine learning calibrations can be biased by the training data, affecting downstream analyses.
method The paper examines simulation-based and data-based calibrations, highlighting their prior dependence and proposing solutions.
result A recently proposed Gaussian Ansatz approach can avoid some biases in simulation-based calibrations.
CPCR mitigates bias in PCR for overparameterized models.
problem Bias in Principal Component Regression (PCR) for overparameterized models.
method Calibrated Principal Component Regression (CPCR) learns a low-variance prior in the PC subspace and calibrates the model in the original feature space.
result CPCR outperforms standard PCR in overparameterized settings, improving prediction across multiple problems.
Reduces false positives in classifying rare online platforms.
problem Challenges in accurately identifying rare online platforms with ML.
method Calibrated probabilities and ensembles to reduce bias.
result Significantly reduces false positives in rare event detection.
Unified framework suppresses model bias in semi-supervised learning with decoupled sampling control.
problem Class imbalance in semi-supervised learning, especially with distributional mismatches.
method Unified framework SC-SSL with decoupled sampling control, explicit expansion capability, and adaptive sampling probabilities.
result Consistent and state-of-the-art performance across various benchmark datasets and distribution settings.
New method corrects selection bias in complex models.
problem Selection bias in statistical studies leading to systematic distortions.
method Amortized Bayesian inference with neural posterior estimation.
result Recover well-calibrated posterior distributions across diverse selection mechanisms.
In classification problems, sampling bias between training data and testing data is critical to the ranking performance of classification scores. Such bias can be both unintentionally introduced by data collection and intentionally introduced by the algorithm, such as under-sampling or weighting techniques applied to i…
Feedback loops amplify dataset biases, affecting future model performance.
problem Feedback loops amplify biases in datasets, risking future model reliability.
method Formalized system where model interactions are recorded and reused, analyzed for bias amplification.
result Models that behave like samples from the training distribution are more stable and calibrated.
Paper proposes an efficient method for calibrating spatio-temporal forecasts.
problem Real-world spatio-temporal forecasting challenges like signal anomalies and distributional shifts.
method Learning with Calibration (ST-TTC) for real-time bias correction.
result ST-TTC improves spatio-temporal forecasting accuracy with reduced computational cost.
Paper estimates optimal classification error with soft labels and calibration.
problem Estimating the optimal classification error with soft labels and calibration.
method Extends previous work on soft labels to estimate Bayes error, addressing bias and corrupted labels.
result The method provides a statistically consistent estimator of the Bayes error, even with imperfectly calibrated soft labels.
FairCal improves face verification accuracy while making results fairer.
problem Bias in face recognition models disproportionately affects minority groups.
method Post-training approach that builds fairer decision classifiers using pre-trained model features.
result State-of-the-art results with increased accuracy and fairness.
This paper addresses measurement errors in high-dimensional compositional data using a log-contrast model calibration approach.
problem Measurement errors in high-dimensional regression models involving compositional covariates.
method Calibration approach for the linear log-contrast model under lenient sparsity conditions.
result Established asymptotic normality of the estimator for inference.
Study shows non-systematic bias in customer satisfaction surveys limits data value.
problem Non-systematic bias in customer satisfaction surveys limits data value.
method Used real customer satisfaction survey data of a large retail bank to show the irreducible error and suggest thoughtful survey design methods.
result A thoughtful survey design can reduce non-systematic error in customer satisfaction surveys.
Label shift refers to the phenomenon where the prior class probability p(y) changes between the training and test distributions, while the conditional probability p(x|y) stays fixed. Label shift arises in settings like medical diagnosis, where a classifier trained to predict disease given symptoms must be adapted to sc…
New algorithm samples Bayesian neural networks for improved calibration.
problem Improving calibration of Bayesian neural networks.
method Symmetric Minibatch Splitting-UBU (SMS-UBU) algorithm.
result SMS-UBU provides better calibration performance than standard methods.
The study examines methods to correct measurement error in nutritional epidemiology studies.
problem Measurement error in nutritional studies leads to biased and underconfident estimates.
method The article reviews various bias-correction models for exposure variables in nutritional epidemiology.
result Bias-correction methods are essential for accurate inference in nutritional studies.
GP-CATE calibrates CATE intervals in few-placebo trials with Gaussian processes.
problem Calibrating uncertainty intervals for CATE in small-arm trials.
method GP-CATE uses Gaussian processes to model each arm's outcome surface directly.
result GP-CATE achieves calibrated coverage where other methods fail.
Proposes a new calibration error estimator for deep neural networks.
problem Improves calibration of deep neural networks, especially for canonical calibration.
method Uses a Dirichlet kernel density estimate to create a low-bias, trainable calibration error estimator.
result Asymptotically converges to true Lp calibration error, enabling efficient estimation and mini-batch updates. New method calibrates machine learning models with theoretical guarantees.
problem Lack of theoretical guarantees for recalibration in multiclass classification.
method PAC-Bayes analysis for generalization error in calibration.
result First optimizable upper bound for generalization error in calibration.
Response calibration is the process of inferring how much the measured data depend on the signal one is interested in. It is essential for any quantitative signal estimation on the basis of the data. Here, we investigate self-calibration methods for linear signal measurements and linear dependence of the response on th…
The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity acros…
Improves robustness of propensity score estimators in challenging settings.
problem Limited overlap, small sample sizes, or unbalanced data.
method Extends calibration techniques for propensity score models, focusing on sample-splitting schemes.
result Calibration reduces variance and bias in inverse probability weighting and double/debiased machine learning frameworks.
Supporting model interpretability for complex phenomena where annotators can legitimately disagree, such as emotion recognition, is a challenging machine learning task. In this work, we show that explicitly quantifying the uncertainty in such settings has interpretability benefits. We use a simple modification of a cla…
Ensembles improve classifier performance by reducing bias, not variance.
problem Improving classifier performance through ensemble methods.
method Extended bias-variance decomposition for classification tasks, introducing dual reparameterization.
result Ensembling reduces bias in classifiers, contrary to the traditional view.
Corrects mismatch in consistency of nuisance estimators for doubly robust methods.
problem Mismatch in consistency of nuisance estimators in doubly robust methods.
method Calibrated debiased machine learning (calibrated DML) with isotonic regression adjustment.
result Calibrated DML yields doubly robust asymptotic normality with slower convergence of nuisance estimators.
It is often observed that the probabilistic predictions given by a machine learning model can disagree with averaged actual outcomes on specific subsets of data, which is also known as the issue of miscalibration. It is responsible for the unreliability of practical machine learning systems. For example, in online adve…
A method corrects bias in estimating a high-dimensional classification rule using auxiliary outcomes.
problem Bias in estimating a high-dimensional classification rule using only one outcome.
method Robust transfer learning approach combining MTL and calibration steps.
result Final estimator achieves lower error than using only the target outcome.
BC-ACI corrects time series forecast bias, improving prediction intervals.
problem Persistent bias in time series forecasts leads to overly conservative prediction intervals.
method Augments ACI with an EWM estimate of forecast bias to correct nonconformity scores and re-center intervals.
result Reduces Winkler interval scores by 13-17% under distribution shifts, improving calibration.
Extends post-prediction inference method for more accurate AI/ML data analysis.
problem Naively using AI/ML predictions as true observations leads to biased results.
method Extends Wang et al. method to relax assumptions and incorporate a scaling factor.
result Yields unbiased point estimates and proper coverage in simulations.
New method preserves GCM spatial dependencies for better climate projections.
problem Systemic biases in GCM output and loss of spatial/temporal dependencies.
method SPECD approach using Vecchia approximation and semi-parametric quantile regression.
result SPECD preserves key marginal and joint distribution properties of precipitation and temperature.
Semi-supervised learning debiased for better performance.
problem Bias in SSL methods impairs theoretical guarantees and empirical performance.
method Simple debiasing approach to remove risk estimate bias.
result Debiased SSL methods provide better calibrated models.
Proposes a new algorithm to estimate invariant subspaces across multilayer networks.
problem Estimating invariant subspaces across heterogeneous multiple networks.
method Bias-corrected joint spectral embedding algorithm that recursively calibrates diagonal bias and iteratively updates the subspace estimator.
result Established entrywise subspace perturbation bound and entrywise eigenvector central limit theorem for the algorithm.
New method corrects seasonal Arctic sea ice predictions with probabilistic models.
problem Systematic biases and errors in climate model forecasts of Arctic sea ice.
method Conditional Variational Autoencoder model to map observation distribution given biased model predictions.
result Probabilistic adjusted forecasts are better calibrated and have smaller errors.
Ask-n-Learn uses gradient embeddings for active learning in image classification.
problem Efficiently labeling large amounts of training data for deep models.
method Gradient embeddings based on pseudo-labels, prediction calibration, and data augmentation.
result Significant improvements over state-of-the-art baselines on image classification tasks.