Cardinal scores (numeric ratings) collected from people are well known to suffer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores. This approach, however, often fares poorly because people's miscalibrations are …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “miscalibration”
Bayesian Consensus: Consensus Estimates from Miscalibrated Instruments under Heteroscedastic Noiseq-fin.ST
Bayesian consensus improves accuracy of forecasts from miscalibrated sources.
problem Aggregating predictions from miscalibrated and noisy sources.
method Bayesian approach to adjust for bias and noise, using hierarchical models.
result Bayesian consensus estimator is unbiased and more efficient than alternatives.
ECCIT improves conditional independence tests by calibrating for miscalibration.
problem Inaccurate frequentist guarantees in CITs, especially in small samples and misspecified models.
method Empirically Calibrated Conditional Independence Tests (ECCIT) that optimize and correct for miscalibration.
result ECCIT achieves valid FDR with higher power than existing calibration strategies.
The paper studies entropy calibration in language models and finds that miscalibration improves slowly with scale.
problem The problem is whether language model entropy calibration improves with scale and if it's possible to calibrate without reducing log loss.
method The authors study a simplified theoretical setting to characterize miscalibration scaling behavior and measure it empirically in language models ranging from 0.5B to 70B parameters.
result The observed scaling behavior of miscalibration is similar to theoretical predictions, indicating slow improvement with scale. The authors also prove theoretically that it is possible to reduce entropy while preserving log loss if access to a black box predicting future entropy is available.
New framework tackles deep learning issues like local traps and miscalibration.
problem Local traps and miscalibration in deep neural networks.
method Sparse deep learning framework with prior annealing algorithms.
result Proposed method successfully addresses local traps and miscalibration.
The paper calibrates uncertainty in dropout variational inference models.
problem Miscalibration of model uncertainty in dropout variational inference.
method Logit scaling methods are extended to recalibrate model uncertainty.
result Logit scaling reduces miscalibration, improving reliability of predictions.
Model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. The uncertainty does not represent the model error well. In this paper, temperature scaling is extended to dropout variational inference to calibrate model uncertainty. Expected uncertainty calibration erro…
Calibrated models can lead to miscalibrated aggregations in strategic interactions.
problem Miscalibration in aggregated predictions from multiple calibrated models.
method Analysis of strategic interactions between calibrated predictors, proving conditions for miscalibration and comparing VCG and Brier-score aggregation methods.
result VCG aggregation method outperforms Brier-score in strategic settings, providing robustness and comparable accuracy.
Hallucinations in models are mislinked estimates, not errors.
problem Hallucinations in generative models as failures to link estimates to plausible causes.
method Formalized hallucinations, showed even optimal estimators hallucinate, provided a general lower bound on hallucinate rate, reframed hallucination as structural misalignment, and experimentally supported theory.
result Hallucinations are structural misalignments between loss minimization and human-acceptable outputs, leading to estimation errors.
Proposes h-calibration for improving miscalibrated probability outputs of neural networks.
problem Improving reliability of probability outputs from neural networks.
method Probabilistic learning framework for calibration, including a simple yet effective post-hoc algorithm.
result Significantly better performance than traditional methods, validated by experiments.
Graph Neural Networks (GNNs) have proven to be successful in many classification tasks, outperforming previous state-of-the-art methods in terms of accuracy. However, accuracy alone is not enough for high-stakes decision making. Decision makers want to know the likelihood that a specific GNN prediction is correct. For …
We study the calibration of several state of the art neural machine translation(NMT) systems built on attention-based encoder-decoder models. For structured outputs like in NMT, calibration is important not just for reliable confidence with predictions, but also for proper functioning of beam-search inference. We show …
Focal loss improves deep neural networks' accuracy and calibration.
problem Miscalibration in deep neural networks.
method Using focal loss and temperature scaling to improve model calibration.
result Focal loss leads to state-of-the-art calibrated models without sacrificing accuracy.
Updating predictive models introduces bias, leading to miscalibration.
problem Bias introduced when updating predictive models after interventions.
method Proposed a causal framework and partially observed Markov decision process.
result Successive predictive scores may converge to undesirable outcomes.
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.
Plots show miscalibration directly as slopes of secant lines.
problem Detecting discrepancies between probabilistic predictions and actual outcomes.
method Cumulative differences between observed and expected values displayed as slopes of secant lines.
result Directly shows miscalibration without binning or kernel density estimation.
LLMs overestimate stock returns and are less accurate at predicting extreme outcomes.
problem Behavioral biases in LLMs' stock return forecasts.
method Comparison of LLM forecasts with crowd-sourced estimates and historical data.
result LLMs overestimate stock returns and are less accurate at predicting extreme outcomes.
Trustworthy Predictive Distributions for Tail Events with Semiparametric Diagnostic Transport Mapsstat.ML
Paper improves predictive distributions for rare events using a simple framework.
problem Local miscalibration of predictive distributions for rare events.
method Semiparametric diagnostic transport maps to correct tail probabilities.
result Semiparametric maps improve predictions for severe weather hazards.
Generative models often misrepresent class frequencies; this paper calibrates them.
problem Miscalibration of class frequencies in generative models.
method Formulated as constrained optimization, using surrogate objectives to approximate constraints.
result Significant reduction in calibration error across various models and applications.
Combines trial and observational data to improve policy evaluation.
problem External validity of randomized trial results in target populations.
method Uses covariate data to model trial sampling and certifies policy evaluations.
result Valid trial-based policy evaluations under model miscalibration.
Discriminative calibration: Check Bayesian computation from simulations and flexible classifierstat.ML
Proposes a new method to improve Bayesian computation accuracy using flexible classification.
problem Bayesian computations accuracy check using rank-based simulation-based calibration has limitations.
method Replaces marginal rank test with a flexible classification approach that learns from data.
result Improves statistical power and provides an interpretable divergence measure of miscalibration.
Develops a method to continuously audit black-box conditional quantile forecasts.
problem Continuous monitoring of black-box forecasts under changing data streams and regimes.
method Distribution-free and game-theoretic testing framework for non-i.i.d. losses.
result Derives finite-time detection guarantees for miscalibrated forecasts based on features.
Novel framework for unbiased confidence estimates in object detection.
problem Unbiased confidence estimates for safety-critical object detection.
method Combines regression output with additional information for calibration.
result Calibrated confidence estimates for image location and scale.
Paper proposes a deep RL method for hedging variable annuities, outperforming misspecified models.
problem Model miscalibration in variable annuity contracts with GMMB and GMDB riders.
method Two-phase deep reinforcement learning approach: training phase in a controlled environment, online learning phase in real market.
result Trained reinforcement learning agent hedges equally well as correct Delta in training phase and outperforms misspecified Deltas.
A new method combines VI and IS to improve Bayesian inference accuracy.
problem Bayesian inference often underestimates posterior tails, leading to miscalibration and degeneracy.
method Proposes a novel combination of optimization and sampling techniques using the forward KL divergence.
result The method guarantees asymptotic consistency and fast convergence to optimal IS and variational approximations.
New methods for scoring function decomposition improve forecast evaluation.
problem Improving forecast evaluation and understanding forecast components.
method Linear recalibration of forecasts for miscalibration, discrimination, and uncertainty.
result Enhanced statistical power and deeper insights into forecast components.
This paper calibrates Gaussian process predictive distributions for Bayesian optimization to improve sampling decisions.
problem Lower-tail miscalibration in GP predictive distributions affects BO sampling decisions.
method Introduces goal-oriented calibration for GP predictive distributions below a threshold .
result Post-hoc method tcGP improves lower-tail calibration and BO performance.
New methods reduce bias in estimating calibration error.
problem Reducing bias in estimating calibration error.
method Synthesizing model outputs and using equal-mass bins.
result Two reliable calibration-error estimators found: debiased estimator and ECE_sweep.
This paper investigates uncertainty calibration in multimodal large language models.
problem Challenges in properly calibrating uncertainty in multimodal large language models.
method Investigation of representative MLLMs across various scenarios, including visual fine-tuning and multimodal training.
result MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments.
New method improves calibration in multi-output probabilistic models.
problem Challenges in achieving multivariate calibration in multi-output regression.
method General regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions.
result Significant improvement in calibration across all pre-rank functions without sacrificing predictive accuracy.
Improves forecast calibration for extreme events using modified loss functions.
problem Improperly specified models do not issue calibrated forecasts for extreme events.
method Adapting loss functions based on weighted scoring rules and tail miscalibration regularization.
result Calibrated forecasts for extreme wind speeds can be improved by suitable adaptations to the loss function during model training.
The wide adoption of Convolutional Neural Networks (CNNs) in applications where decision-making under uncertainty is fundamental, has brought a great deal of attention to the ability of these models to accurately quantify the uncertainty in their predictions. Previous work on combining CNNs with Gaussian processes (GPs…
Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to …
DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Trainingstat.ML
We study optimal covariate balance for causal inferences from observational data when rich covariates and complex relationships necessitate flexible modeling with neural networks. Standard approaches such as propensity weighting and matching/balancing fail in such settings due to miscalibrated propensity nets and inapp…
The role of uncertainty quantification (UQ) in deep learning has become crucial with growing use of predictive models in high-risk applications. Though a large class of methods exists for measuring deep uncertainties, in practice, the resulting estimates are found to be poorly calibrated, thus making it challenging to …
Probabilistic classifiers output a probability distribution on target classes rather than just a class prediction. Besides providing a clear separation of prediction and decision making, the main advantage of probabilistic models is their ability to represent uncertainty about predictions. In safety-critical applicatio…
Online Platt Scaling adapts to varying data distributions.
problem Adapting Platt scaling to non-i.i.d. settings with distribution drift.
method Combines Platt scaling with online logistic regression and calibeating.
result OPS+calibeating method is guaranteed to be calibrated for adversarial outcomes.
DRO-NPE improves neural posterior estimation by reducing overconfidence and overfitting.
problem Overconfident and unreliable posteriors in simulation-based inference with limited simulation budgets.
method Distributionally robust approach using Wasserstein ambiguity set and KL-based metrics.
result Consistently improves coverage and calibration across benchmark tasks.
New decision-theoretic calibration error metric improves prediction reliability.
problem Improving the reliability of predictions for decision-making.
method Proposed Calibration Decision Loss (CDL) and an efficient algorithm to achieve near-optimal CDL.
result Near-optimal CDL guarantees vanishing payoff loss from miscalibration.
CP4SBI improves the calibration of credible sets in SBI models.
problem Inaccurate credible sets in SBI models lead to underestimation of true parameters.
method Develops a local conformal calibration framework for SBI models.
result Improves the quality of uncertainty quantification for neural posterior estimators.
We identify a new variational inference scheme for dynamical systems whose transition function is modelled by a Gaussian process. Inference in this setting has either employed computationally intensive MCMC methods, or relied on factorisations of the variational posterior. As we demonstrate in our experiments, the fact…
A new method calibrates Gaussian processes for more accurate uncertainty estimates.
problem Uncertainty estimates from Gaussian processes are often miscalibrated in practice.
method A novel calibration approach using different hyperparameters to generate more accurate predictive quantiles.
result The method yields tighter predictive quantiles and is more flexible than existing approaches.
Randomized predictions ensure fair and accurate individual calibration in machine learning.
problem Systematic bias in typical calibration methods leads to unfair predictions for certain subgroups.
method Randomization of predictions to enforce individual calibration, trading off bias with variance.
result Randomized regression functions are more calibrated for arbitrary subgroups and achieve higher utility.
Unified Uncertainty Calibration improves AI predictions by combining different types of uncertainty.
problem AI classifiers struggle with uncertainty, leading to miscalibrated predictions and poor performance.
method Unified Uncertainty Calibration (U2C) combines aleatoric and epistemic uncertainties to improve prediction quality.
result U2C outperforms traditional reject-or-classify methods across various ImageNet benchmarks.
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
A new metric CKCE improves model calibration comparison.
problem Comparing the calibration of probabilistic models is challenging.
method CKCE based on Hilbert-Schmidt norm of conditional mean operators.
result CKCE provides more consistent and robust model calibration comparisons.
New method calibrates classifier probabilities with guaranteed coverage.
problem Inaccurate probability estimates by classifiers in high-risk applications.
method Adaptive temperature scaling algorithm for conformal prediction.
result Improves calibration error measures and standard metrics across various tasks.
Calibration without labels in multiple testing
problem Interpretable error probabilities in large-scale hypothesis testing
method Constructing pseudo-labels from spacings of ordered -values
result Finding that -value can be severely miscalibrated