Exact distribution of split conformal prediction coverage found.
problem Determining the reliability of prediction sets in batch mode.
method Analysis of exchangeable data to find universal distribution of empirical coverage.
result Exact distribution of empirical coverage is universal and determined by nominal miscoverage level and calibration sample size.
Study non-monotonic loss functions in CRC, achieving valid risk control with large calibration samples.
problem Non-monotonic loss functions in CRC, violating existing theory's monotonicity assumption.
method Finite grid selection, calibration sample size analysis, Lipschitz continuity, monotonicity, distribution shift.
result Valid CRC achieved with large calibration samples, optimal excess risk rate of log ( m ) / n \sqrt{\log(m)/n} log ( m ) / n . Improves robustness of propensity score estimators in challenging settings.
problem Limited overlap, small sample sizes, or unbalanced data.
method Extends calibration techniques for propensity score models, focusing on sample-splitting schemes.
result Calibration reduces variance and bias in inverse probability weighting and double/debiased machine learning frameworks.
New calibration bands for various distributions improve testing for auto-calibration.
problem Testing for auto-calibration in finite samples is challenging.
method Construct calibration bands for the exponential dispersion family using finite sample properties.
result Calibration bands allow for various tests for calibration and auto-calibration.
Tabular foundation models outperform other methods in conditional density estimation across various datasets.
problem Estimating the full conditional distribution of a response given tabular covariates, especially in settings with heteroscedasticity, multimodality, or asymmetric uncertainty.
method Benchmarked three tabular foundation model variants (TabPFN and TabICL) against six CDE baselines on 39 real-world datasets.
result Tabular foundation models achieve the best CDE loss, log-likelihood, and CRPS across all sample sizes, outperforming other methods.
Conditional independence testing is a fundamental problem underlying causal discovery and a particularly challenging task in the presence of nonlinear and high-dimensional dependencies. Here a fully non-parametric test for continuous data based on conditional mutual information combined with a local permutation scheme …
New methods improve confidence set calibration in complex models.
problem Challenges in maintaining confidence set coverage in complex models.
method TRUST and TRUST++ methods using simulated data for calibration.
result Methods achieve distribution-free conditional coverage and robust inference.
New algorithm tests model calibration in nearly-linear time.
problem Testing model calibration from samples efficiently.
method Reformulated as minimum-cost flow, solved with dynamic programming.
result Optimal testing problem solved in nearly-linear time.
The paper proposes a method to produce well-calibrated predictions in regression tasks using maximum mean discrepancy.
problem The need for accurate uncertainty quantification in machine learning predictions.
method The method uses maximum mean discrepancy to minimize the kernel embedding measure and calibrate predictions.
result The method produces well-calibrated and sharp prediction intervals, outperforming state-of-the-art methods.
Study shows pooling scores for conformal prediction distorts group coverage.
problem Pooling scores for conformal prediction distorts group coverage.
method Derived conservation law and lower bound, demonstrated tension between fairness definitions, quantified trade-off between policies.
result Pooling scores for conformal prediction distorts group coverage.
A novel Hawkes Process model captures order sizes in LOBs, improving fit quality and market impact studies.
problem Capturing the variability in order sizes in Limit Order Books (LOBs).
method Compound Hawkes Process with time-varying parameters and non-parametric calibration.
result Improved fit quality and empirical market impact function replication.
JUCAL jointly calibrates aleatoric and epistemic uncertainties in classifier ensembles.
problem Misrepresentation of predictive uncertainty due to unbalanced aleatoric and epistemic uncertainties.
method Joint Uncertainty Calibration (JUCAL) that jointly calibrates two constants to weight and scale uncertainties.
result Significantly outperforms state-of-the-art calibration methods across various text classification tasks.
Study robustness of split conformal prediction under adversarial attacks.
problem Ensuring distribution-free coverage guarantees in CP under adversarial conditions.
method Theoretical analysis and extensive experiments on split conformal prediction robustness.
result Prediction coverage varies with calibration-time attack strength, enabling control over coverage under adversarial tests.
The study examines how much data is needed for generative and vision-language models to make reliable predictions.
problem Ensuring reliable predictions with low data for models used in medical decision support.
method Analyzes uniform convergence bounds for VLM-induced classifiers under low-dimensional semantic representations.
result Finite-sample uniform convergence bounds for accuracy and calibration functionals of VLM-induced classifiers.
ECCIT improves conditional independence tests by calibrating for miscalibration.
problem Inaccurate frequentist guarantees in CITs, especially in small samples and misspecified models.
method Empirically Calibrated Conditional Independence Tests (ECCIT) that optimize and correct for miscalibration.
result ECCIT achieves valid FDR with higher power than existing calibration strategies.
Calibrated probabilistic solvers improve accuracy of ODE estimates.
problem Uncertainty in probabilistic ODE solutions is not well-calibrated for adaptive step sizes.
method Introduce and assess several calibration methods for probabilistic ODE solvers.
result Calibration methods interact efficiently with adaptive step-size selection, improving posteriors.
A new method calculates intrinsic effective sample size for manifold-valued data.
problem Challenges in choosing effective sample size for manifold-valued data.
method Proposes an intrinsic effective sample size based on kernel discrepancy.
result Establishes an exact finite-sample risk interpretation and consistency of the estimator.
The paper improves conformal prediction by analyzing the beta law of conditional coverage.
problem Improving finite-sample marginal coverage guarantees for non-i.i.d. data.
method The method uses Wasserstein distances to quantify deviations from the beta law of conditional coverage.
result The framework provides direct bounds on marginal coverage gaps and bad-calibration probabilities.
Proposes a method to calibrate data for more accurate linear correlation testing.
problem Inaccurate Pearson's correlation coefficient due to sample size and data non-normality.
method Predictive data calibration using machine learning to condition data on expected linear relationship.
result Calibrated Pearson's correlation coefficient yields a calibrated p-value and r estimate for posterior probability interpretation.
Study on computing and estimating calibration distance, showing hardness and efficiency.
problem Computing and estimating calibration distance under different assumptions.
method Efficient algorithm for exact computation, polynomial-time approximation scheme; sample-based estimation for upper bounds.
result The problem becomes NP-hard when assumptions are removed, but efficient algorithms exist under certain conditions.
Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliabil…
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
problem Numerical instability and slow progress in Q-learning and SARSA due to step-size calibration.
method Reformulate iterative updates as fixed-point equations, scaling step-sizes inversely with feature norms.
result Implicit methods maintain stability over broader step-size ranges and achieve comparable convergence rates.
Extend CPS to non-exchangeable settings with observation-specific permutation weights
problem Calibrated predictive bands under distributional shifts
method Encoding distributional shifts through observation-specific permutation weights
result Shift-aware predictive systems remain valid
Develops geometric framework for uncertainty-aware multi-class classification.
problem Silent failure of AI models when uncertain, especially in multi-class settings.
method Geometric framework treating probability vectors as points on the ( c − 1 ) (c-1) ( c − 1 ) -dimensional probability simplex, using Fisher--Rao metric for calibration and uncertainty quantification. result Empirical validation shows 72.5% of errors captured while deferring 34.5% of ambiguous predictions, reducing automated decision error rates from 16.8% to 6.9%.
Observing prices of European put and call options, we calibrate exponential Lévy models nonparametrically. We discuss the efficient implementation of the spectral estimation procedures for Lévy models of finite jump activity as well as for self-decomposable Lévy models. Based on finite sample variances, confidence inte…
We study the out-of-sample properties of robust empirical optimization problems with smooth φ φ φ -divergence penalties and smooth concave objective functions, and develop a theory for data-driven calibration of the non-negative "robustness parameter" δ δ δ that controls the size of the deviations from the nominal model. Bu…
The paper investigates how dataset quality and heterogeneity affect model confidence in machine learning.
problem Understanding how dataset quality and heterogeneity impact model confidence in machine learning.
method The study uses theoretical explanations and experimental demonstrations to investigate the effects of dataset size, label noise, and class heterogeneity on model confidence.
result Label noise reduces model confidence, while reduced dataset size increases it, and class heterogeneity leads to inconsistent confidence across classes.
Study shows uncertainty calibration improves BO performance, but not as much as model type.
problem Effect of model uncertainties on Bayesian optimization performance.
method Extensive study comparing different surrogate models and their uncertainty calibration.
result Gaussian Processes outperform other models in BO, and uncertainty calibration does not significantly improve regret.
Improves risk control in predictions using semi-supervised calibration.
problem Noisy hyper-parameter tuning from limited labeled data.
method Semi-supervised calibration using unlabeled data to tune hyper-parameters rigorously.
result Improves prediction accuracy without sacrificing statistical validity.
New method detects changes in high-dimensional data from small samples.
problem Detecting changes in high-dimensional data with limited samples.
method Angular kernel scan framework for detecting marginal distributional shifts.
result Exact population mean factorization and asymptotically distribution-free test.
Spectral Adaptive Conformal Prediction for Structured Non-Exchangeable Data
problem Improving prediction intervals for non-exchangeable time-indexed datasets
method Spectral adaptive conformal prediction
result Improves on fixed spectral weighting while monitoring uncertainty changes
Meta-learning reduces set prediction size in conformal prediction for few-shot calibration.
problem Inefficient set prediction in conformal prediction for limited training data.
method Meta-learning approach using cross-validation-based conformal prediction.
result Meta-learning scheme reduces set prediction size and preserves formal guarantees.
Adaptive method improves prediction intervals with global coverage guarantees and local error distribution.
problem Global coverage guarantees of conformal regression are often violated by local error distributions.
method Adaptive Conformal Regression with Jackknife+ Rescaled Scores
result Improves local coverage without sacrificing global coverage, especially in low-data regimes.
Causal discovery can be a powerful tool for investigating causality when a system can be observed but is inaccessible to experiments in practice. Despite this, it is rarely used in any scientific or medical fields. One of the major hurdles preventing the field of causal discovery from having a larger impact is that it …
New method improves model calibration by adjusting confidence based on prediction correctness.
problem Improving model confidence alignment with true class probabilities.
method Post-hoc calibration objective using transformed samples for training.
result Competitive calibration performance on in-distribution and out-of-distribution test sets.
MCP extends conformal prediction to vector-valued score functions without data splitting.
problem Fixed prediction set shapes in scalar score functions limit coverage guarantees.
method MCP uses a single optimization problem for prediction set design and calibration, eliminating data splitting.
result RemMCP and RelMCP achieve target coverage with smaller or comparable prediction set sizes, reducing variance.
Develops a calibration prediction interval for non-parametric regression and neural networks.
problem Lack of accurate conditional prediction in regression settings.
method Calibration Prediction Interval (cPI) using Deep Neural Networks (DNN) or kernel methods.
result Asymptotically valid coverage rate and high probability of coverage rate with large sample sizes.
Unified framework for generalized Venn and Venn-Abers calibration for reliable prediction.
problem Asymptotic guarantees of popular distribution-free methods in model calibration.
method Unified framework extending Vovk's approach to generic loss functions, transforming predictors into set-valued predictions.
result Finite-sample set predictions shrink to a single conditionally calibrated prediction, capturing epistemic uncertainty.
Paper presents unsupervised calibration for split conformal classification.
problem Inconvenient requirement of labeled calibration samples.
method Uses unsupervised calibration samples alongside supervised training samples.
result Achieves comparable performance to supervised calibration methods.
TS improves class coverage but reduces CP set size, offering a trade-off.
problem Combining temperature scaling with conformal prediction for deep classifiers.
method Empirical study and mathematical theory of TS's effect on CP.
result TS allows trading prediction set size and conditional coverage.
CJE calibrates cheap LLM judges against an oracle, achieving high accuracy at a fraction of the cost.
problem Inexpensive LLM judges can produce biased rankings, leading to unreliable outcomes.
method CJE uses a small oracle to calibrate cheap scores, then evaluates at scale with valid uncertainty.
result CJE achieves 99% pairwise ranking accuracy at 14x lower cost compared to a 16x oracle/judge cost ratio.
Complex classification performance metrics such as the F β {}_β β -measure and Jaccard index are often used, in order to handle class-imbalanced cases such as information retrieval and image segmentation. These performance metrics are not decomposable, that is, they cannot be expressed in a per-example manner, which hinder…
Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popu…
This paper introduces minimum-risk recalibration for probabilistic classifiers, improving their reliability and accuracy.
problem Improving the reliability and accuracy of probabilistic classifiers.
method Minimum-risk recalibration within the MSE decomposition framework, analyzing UMB method and label shift adaptation.
result The optimal number of bins for UMB scales with n 1 / 3 n^{1/3} n 1/3 , resulting in a risk bound of approximately O ( n − 2 / 3 ) O(n^{-2/3}) O ( n − 2/3 ) . New algorithms achieve decision calibration without sample complexity dependent on feature dimension.
problem Achieving decision calibration for nonlinear loss functions with polynomial sample complexity.
method Developed smooth relaxation of decision calibration, enabling dimension-free algorithms.
result Efficient algorithms post-process predictors to satisfy decision calibration without worsening accuracy.
Proposes PFWCP for multi-agent tasks with privacy and validity guarantees.
problem Challenges in uncertainty quantification for multi-agent settings.
method Personalized federated weighted conformal prediction (PFWCP) combining local density ratio weighting and weighted quantile aggregation.
result Asymptotically valid coverage guarantees for each agent in heterogeneous settings.
A new framework evaluates LLM calibration in open-ended QA.
problem Evaluating LLM calibration in open-ended QA settings.
method Sem-ECE framework: sampling answers, grouping by semantic classes, and using frequencies as confidence.
result Sem-ECE estimators are unbiased and Sem 2 _2 2 achieves smaller calibration error. New algorithms find optimal policies without knowing MDP span.
problem Finding optimal policies in MDPs without knowing span.
method Horizon calibration and span penalization techniques.
result First algorithms achieving optimal span-based complexity without prior knowledge.