The study examines when to trust confidence thresholding in pseudo-labelling regression.
problem Calibrated probabilities from classifiers used for pseudo-labelling need careful handling to avoid bias in downstream regression.
method Developed a diagnostic apparatus to predict and bound the bias induced by confidence thresholding, derived a closed-form expression for the attenuation bias.
result The bias can be predicted from the residual score variance V∗, motivating a structural separation between classifier features and downstream controls. This paper calibrates Gaussian process predictive distributions for Bayesian optimization to improve sampling decisions.
problem Lower-tail miscalibration in GP predictive distributions affects BO sampling decisions.
method Introduces goal-oriented calibration for GP predictive distributions below a threshold t. result Post-hoc method tcGP improves lower-tail calibration and BO performance.
Improves graph recovery in Gaussian graphical modeling.
problem Calibrating regularization parameters for graph recovery.
method Thresholded adaptive validation applied to graphical lasso.
result Thresholding pipeline improves graph recovery.
New method provides calibrated feature importance explanations for regression models.
problem Lack of uncertainty quantification in existing local explanation methods.
method Extension of Calibrated Explanations method to support regression and probabilistic regression.
result Calibrated Explanations for regression provides quantified uncertainty and robust explanations.
Noise-Aware Conformal Prediction (NACP) calibrates CP for noisy labels.
problem Calibrating Conformal Prediction with noisy labels.
method Estimate conformal threshold from noisy labels using uniform noise coverage guarantee.
result Finite sample coverage guarantee for uniform noise remains effective in high-class tasks.
New method calibrates false detection rates in sequential change detection.
problem Challenges in setting time-invariant thresholds for false positives.
method Simulation-based approach to time-varying thresholds.
result Accurately targets desired expected runtime while keeping false positive rate constant.
Framework for precise recall control in spatial conflation tasks.
problem Precise recall control in large-scale spatial conflation tasks to avoid downstream analytics failures and excessive manual review.
method End-to-end framework using equigrid bounding-box filter, CSR representation, neural ranker, and inverse-variance weighted ensemble of threshold estimators.
result Achieves exact recall with sub-percent variance over tens of millions of geometry pairs, runs on a single TPU v3 core.
In high-dimensional classification settings, we wish to seek a balance between high power and ensuring control over a desired loss function. In many settings, the points most likely to be misclassified are those who lie near the decision boundary of the given classification method. Often, these uninformative points sho…
Recently, a novel family of biologically plausible online algorithms for reducing the dimensionality of streaming data has been derived from the similarity matching principle. In these algorithms, the number of output dimensions can be determined adaptively by thresholding the singular values of the input data matrix. …
New method calibrates uncertainty estimates for image classifiers without labeled data.
problem Uncertainty estimates for modern classifiers are unreliable without labeled calibration data.
method Calibrates uncertainty estimates using unlabeled examples for distribution shifts.
result Proposes a method that provides excellent uncertainty estimates under natural distribution shifts.
Fairness audits fail under missing protected labels, especially at zero access.
problem Understanding the reliability of fairness audits with incomplete protected-label data.
method Introduced a seed-calibrated stress test to separate missingness effects from seed-to-seed movement.
result Missing protected labels do not significantly alter fairness mitigation methods, but they can lead to harmful intersectional outcomes.
Prior-weighted logistic regression has become a standard tool for calibration in speaker recognition. Logistic regression is the optimization of the expected value of the logarithmic scoring rule. We generalize this via a parametric family of proper scoring rules. Our theoretical analysis shows how different members of…
New framework assesses LM uncertainty without thresholding.
problem Uncertainty quantification for LMs, especially comparing different measures.
method Rank-Calibration framework to assess uncertainty and confidence measures.
result Higher uncertainty correlates with lower generation quality.
The paper studies calibration in ML models for wireless networks, showing key theoretical and practical insights.
problem Ensuring ML models in wireless networks deliver well-calibrated confidence scores for reliable decision-making.
method Theoretical analysis and simulation-based experiments using Platt scaling and isotonic regression.
result Well-calibrated models improve the system's minimum achievable OP and are part of a broader class of predictors.
The paper develops methods to predict the probability of achieving a user goal in a task, ensuring the system alerts when the probability falls below a threshold.
problem Ensuring an autonomous system achieves the user's goal with calibrated probability estimates.
method Invertible conformal prediction using Probability-space Conformalized Quantile Regression (PCQR) to produce well-calibrated conditional prediction intervals.
result The method produces well-calibrated probabilities that the cumulative reward will fall within a user-specified target interval, with finite-sample guarantees.
Simple online monitor detects unsafe LLM outputs.
problem LLMs generate unsafe outputs despite training.
method Thresholding external verifier signal to decide alarms.
result Simple design competitive with advanced methods.
Researchers calibrate an adaptive Farmer-Joshi model to recover stylized facts in financial markets.
problem Recovering stylized facts in financial markets using the Farmer-Joshi model.
method Calibrated an adaptive Farmer-Joshi model using genetic and Nelder-Mead algorithms, incorporating agent adaptation.
result The adaptive model recovers additional stylized facts, including auto-correlations and kurtosis, compared to the original model.
New method selects recent similar periods for better electricity price forecasting.
problem Improving accuracy in forecasting electricity prices.
method Change-point detection (NOT method) to select calibration periods; estimating autoregressive models only for selected data.
result Significant improvement in forecasting accuracy compared to existing methods.
This work interprets GELU and related activations via a first-order loss function.
problem Understanding and optimizing activation functions in neural networks.
method Complementary interpretation using the Gaussian first-order loss function.
result Calibrated or learned uniform-threshold gates are competitive and often outperform GELU, ReLU, and SiLU/Swish.
Python package 'nonconform' simplifies conformal anomaly detection.
problem Heuristic thresholding in anomaly detection systems.
method 'nonconform' package converts anomaly scores into calibrated p-values.
result Statistically principled anomaly detection is made accessible.
Neural network quantization procedure is the necessary step for porting of neural networks to mobile devices. Quantization allows accelerating the inference, reducing memory consumption and model size. It can be performed without fine-tuning using calibration procedure (calculation of parameters necessary for quantizat…
L-ARC improves model fairness by localizing risk guarantees.
problem Improving model fairness in tasks like image segmentation and wireless networks.
method Localized Adaptive Risk Control (L-ARC) updates a threshold function in RKHS to target localized statistical risk guarantees.
result L-ARC produces prediction sets with improved fairness across different data subpopulations.
Decision makers increasingly rely on algorithmic risk scores to determine access to binary treatments including bail, loans, and medical interventions. In these settings, we reconcile two fairness criteria that were previously shown to be in conflict: calibration and error rate equality. In particular, we derive necess…
New metrics CWSA and CWSA+ improve model evaluation under confidence thresholds.
problem Lack of metrics capturing model reliability under confidence thresholds.
method Introducing CWSA and CWSA+ metrics that reward confident accuracy and penalize overconfident mistakes.
result CWSA and CWSA+ outperform classical metrics in trust-sensitive tests.
The paper models rating transitions and calibrates them to market data for XVA calculations.
problem Calibrating rating models to both historical and market data for accurate XVA calculations.
method Modeling rating transitions as a Markov chain, calibrating to historical and market data, proposing a novel calibration procedure.
result Improved XVA scheme through better calibration of rating models.
New method improves auto-labeling accuracy by optimizing confidence functions.
problem Overconfident model scores lead to poor TBAL performance.
method Developed a new post-hoc method, Colander, to optimize TBAL confidence functions.
result Achieves up to 60% improvement in coverage over baseline methods.
Exact simulation of correlated binary outcomes using PMF constraints and linear programming.
problem Simulating dependent Bernoulli outcomes with specific means and correlations.
method Formulate the problem over the joint Bernoulli PMF, impose constraints, and solve as a linear program. Use convex-hull characterization and truncated-moment completion scheme for feasibility and simulation.
result Exact simulation framework for correlated binary outcomes, providing a convex-hull characterization and truncated-moment completion scheme.
Conformal Bayes under label shift: post-hoc calibration vs. in-training adaptation
problem Bayesian prediction sets under label shift
method Post-hoc calibration vs. In-training adaptation
result Both strategies achieve valid coverage equally in an unbiased training regime
A new framework for consistent segmentation evaluation reduces operating losses.
problem Inconsistent thresholding-based segmentation methods lead to suboptimal solutions.
method Developed a consistent ranking-based framework (RankDice/RankIoU) using Bayes rules and Dice-/IoU-calibration.
result The proposed framework is Dice-/IoU-calibrated and provides excess risk bounds and convergence rates.
Study finds incorporating fairness in healthcare models doesn't improve performance or net benefit.
problem Addressing health inequities in healthcare through algorithmic fairness.
method Empirical case study using models to estimate atherosclerotic cardiovascular disease risk.
result Incorporating fairness considerations into model training objective does not improve model performance or net benefit.
Study shows pooling scores for conformal prediction distorts group coverage.
problem Pooling scores for conformal prediction distorts group coverage.
method Derived conservation law and lower bound, demonstrated tension between fairness definitions, quantified trade-off between policies.
result Pooling scores for conformal prediction distorts group coverage.
Study proves minimality of certain hyperplane intersections in wide cones.
problem Proving minimality of intersections of hyperplanes in wide cones.
method Calibration method adapted for free-boundary variations.
result Existence of a threshold angle for minimality in high dimensions.
A popular methodology for building binary decision-making classifiers in the presence of imperfect information is to first construct a non-binary "scoring" classifier that is calibrated over all protected groups, and then to post-process this score to obtain a binary decision. We study the feasibility of achieving vari…
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, ha…
Algorithmic insurance tackles financial risks from AI errors, proving CVaR-optimal thresholds reduce tail risk.
problem High-stakes AI errors lead to heterogeneous losses, challenging traditional insurance assumptions.
method Analyzed binary classification performance to tail risk exposure, using CVaR to quantify extreme losses.
result CVaR-optimal thresholds reduce tail risk up to 13-fold compared to accuracy maximization.
Decision trees and shallow neural networks have different geometric complexities, impacting their interpretability and accuracy.
problem The geometric simplicity of decision boundaries in decision trees conflicts with the approximation capabilities of shallow neural networks.
method Analysis of the Radon total variation (RTV) seminorm to compare geometric complexity of decision regions and neural network approximations.
result Smooth barrier scores can approximate decision regions with finite RTV, but their performance depends on the tube-mass condition near the decision boundary.
Trimming helps in conformal prediction when it separates anomaly scores.
problem Effectiveness of trimming in conformal prediction under contamination.
method Analyse fixed-threshold trimming as a replacement of the contaminated calibration law with a retained law.
result Trimming helps when it separates anomaly scores, reducing clean-target coverage to a one-dimensional score-CDF transfer problem.
Null-Calibrated Conformal Selection via Target-Membership Scores
problem Identifying test candidates whose unknown responses fall in a target region while controlling the false discovery rate
method Membership-score-based conformal selection
result Finite-sample valid null p-values
Decision-calibrated prediction sets improve power system operations by reducing unnecessary costs.
problem Balancing operating costs and reliability in power systems with renewable uncertainty.
method Learn conditional prediction sets as sub-level sets of norm-based score functions, calibrate uncertainty sets based on reliability of downstream decisions.
result Decision-calibrated sets lead to more efficient operations with smaller uncertainty sets and lower costs compared to standard coverage-based calibration.
MCP extends conformal prediction to vector-valued score functions without data splitting.
problem Fixed prediction set shapes in scalar score functions limit coverage guarantees.
method MCP uses a single optimization problem for prediction set design and calibration, eliminating data splitting.
result RemMCP and RelMCP achieve target coverage with smaller or comparable prediction set sizes, reducing variance.
This paper concerns the problem of matrix completion, which is to estimate a matrix from observations in a small subset of indices. We propose a calibrated spectrum elastic net method with a sum of the nuclear and Frobenius penalties and develop an iterative algorithm to solve the convex minimization problem. The itera…
Comparison of decision curve analysis and cost curves for model evaluation.
problem Evaluating classification performance across different operating contexts.
method Comparison of Decision Curve Analysis (DCA) and Cost Curves.
result DCA and Cost Curves are closely related, with Brier curves being more generally applicable.
Class imbalance presents a major hurdle in the application of data mining methods. A common practice to deal with it is to create ensembles of classifiers that learn from resampled balanced data. For example, bagged decision trees combined with random undersampling (RUS) or the synthetic minority oversampling technique…
Study evaluates three class imbalance techniques across diverse datasets.
problem Class imbalance in binary classification tasks.
method Synthetic Minority Over-sampling Technique (SMOTE), Class Weights tuning, Decision Threshold Calibration.
result Decision Threshold Calibration is the most consistently effective technique.
New insights link no-regret learning to online conformal prediction in adversarial settings.
problem Understanding the relationship between no-regret learning and online conformal prediction in adversarial environments.
method Analysis of existing algorithms and new connections between no-regret learning and conformal prediction.
result No-regret learning algorithms can provide group-conditional coverage guarantees in adversarial settings.
Two approaches improve conformal Bayes for label shift, one post-hoc and one in-training.
problem Improving prediction sets for target domain under label shift.
method Two complementary approaches: post-hoc calibration and in-training adaptation.
result In-training adaptation achieves up to 43% width reduction at unchanged coverage.
The paper addresses fairness issues in screening classifiers, proposing within-group monotonicity to avoid unfair treatment of qualified candidates.
problem Within-group unfairness in screening classifiers using calibrated models.
method Introducing within-group monotonicity as a property to avoid unfair treatment and developing an efficient post-processing algorithm based on dynamic programming.
result Within-group monotonicity can be achieved efficiently and often at a small cost, improving fairness without significantly compromising prediction accuracy.
In this work, we propose data augmentation methods for embeddings from pre-trained deep learning models that take a weighted combination of a pair of input embeddings, as inspired by Mixup, and combine such augmentation with extra label softening. These methods are shown to significantly increase classification accurac…