Proposes new method for calibrating treatment effect predictors.
problem Calibrating predictors of heterogeneous treatment effects.
method Causal isotonic calibration and cross-calibration.
result Achieves fast calibration rates under weak conditions.
Deep ensembles don't necessarily improve calibration in low data regimes.
problem Calibration issues in deep learning models, especially in low data regimes.
method Examination of data-augmentation, ensembling, and post-processing calibration methods.
result Standard ensembling techniques can lead to less calibrated models in low data regimes.
Combining ensembles and data augmentation harms model calibration.
problem Improving model calibration and robustness with ensembles and data augmentation leads to a trade-off.
method Combining ensemble averaging and data augmentation techniques.
result Combining ensembles and data augmentation can harm model calibration.
Better calibration for small datasets improves probability estimates.
problem Limited data hinders traditional calibration methods.
method Generating additional calibration data improves classifier performance.
result The proposed approach enhances calibration for small datasets.
Ensemble models improve prediction calibration for mismatched distributions.
problem Calibration issues in deep neural networks with mismatched train and test distributions.
method Simple data augmentation and mixing techniques for ensemble models.
result Improves calibration and accuracy on CIFAR10 and CIFAR100 benchmarks.
Paper proposes a method to predict MOBA game winners with calibrated confidence.
problem Predicting MOBA game winners with noisy data and uncertain noise.
method A novel confidence-calibration method considering data uncertainty.
result Achieves outstanding expected calibration error (ECE) of 0.57%.
Mix-n-Match improves uncertainty calibration in deep learning.
problem Post-hoc calibration of machine learning classifiers.
method Ensemble and composition strategies to improve accuracy, efficiency, and expressive power.
result Mix-n-Match strategies achieve better data-efficiency and expressive power while maintaining classification accuracy.
New framework allows selective removal of stale data in option calibration.
problem Inability to remove old data from calibrated option pricing models without full retraining.
method Introduces operator-theoretic Gauss-Newton framework for selective forgetting.
result Provides stability guarantees and perturbation bounds for selective data removal.
BRPC online Bayesian calibration handles gradual and abrupt system changes.
problem Aligning model outputs with field observations in evolving systems.
method Bayesian Recursive Projected Calibration (BRPC) for streaming data under simulator mismatch and nonstationarity.
result Improves calibration accuracy under gradual changes and robustness under abrupt regime shifts.
X-CAL improves survival model calibration without sacrificing predictive power.
problem Improving the calibration of survival models to better match observed data.
method Explicit calibration (X-CAL) turns distributional calibration into a differentiable objective for survival modeling.
result X-CAL improves calibration metrics without significantly reducing predictive performance.
Self-calibrating conformal prediction improves interval efficiency and offers a practical alternative.
problem Improving the reliability and uncertainty quantification of machine learning predictions.
method Combines Venn-Abers calibration and conformal prediction for binary and regression problems.
result Improves interval efficiency through model calibration and offers practical alternatives.
The paper highlights how machine learning calibrations can be biased by training data.
problem Machine learning calibrations can be biased by the training data, affecting downstream analyses.
method The paper examines simulation-based and data-based calibrations, highlighting their prior dependence and proposing solutions.
result A recently proposed Gaussian Ansatz approach can avoid some biases in simulation-based calibrations.
New study on neural network calibration, linking it to generalization gap.
problem Neural networks lack strong guarantees on calibration.
method Decomposed calibration error into train set and generalization gap.
result Models with small generalization gap are well-calibrated.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
This work evaluates and benchmarks calibration metrics for data-driven regression models.
problem Conflicting results from different calibration metrics make it hard to compare and interpret model performance.
method Systematically extracted and benchmarked 14 regression calibration metrics across various data types and recalibration methods.
result Many metrics disagree on the same recalibration result, highlighting the need for careful metric selection.
Survey of Optimal Transport for model calibration.
problem Model calibration using Optimal Transport.
method General framework and numerical algorithms for various models.
result Calibration of volatility models and path-dependent options.
New method calibrates DPMs to improve likelihood bounds.
problem Improving the likelihood bounds of DPMs.
method Deriving concentration bounds and using the optional stopping theorem for data scores to calibrate DPMs.
result Calibrated DPMs can increase likelihood bounds and improve sampling quality.
Proposes efficient calibration for indoor localization models.
problem Calibration data scarcity in wireless indoor localization.
method Uses synthetic labels and prediction sets to fine-tune a predictor and estimate bias.
result Yields rigorous coverage guarantees for prediction sets.
The calibration of a measurement device is crucial for every scientific experiment, where a signal has to be inferred from data. We present CURE, the calibration uncertainty renormalized estimator, to reconstruct a signal and simultaneously the instrument's calibration from the same data without knowing the exact calib…
Isotonic regression binning affects calibration statistics of machine learning models.
problem Isotonic regression binning introduces aleatoric uncertainty in calibration statistics.
method Calibration error statistics are recalibrated using isotonic regression, which produces stratified uncertainties.
result Stratified uncertainties lead to significant differences in bin-based calibration statistics.
Bayesian neural networks outperform calibrated neural networks for tabular data.
problem Uncertainty in neural network predictions for tabular data.
method Bayesian neural networks vs. post-hoc calibration methods.
result Bayesian neural networks yield competitive performance compared to calibrated neural networks.
Efficiently calibrates Bergomi models to VIX derivatives using vector quantization.
problem Calibrating Bergomi models to VIX derivatives for accurate pricing.
method Applied vector quantization in mixed Bergomi models for fast and efficient option pricing.
result Calibration of Bergomi models to VIX derivatives is feasible and accurate over daily data.
Response calibration is the process of inferring how much the measured data depend on the signal one is interested in. It is essential for any quantitative signal estimation on the basis of the data. Here, we investigate self-calibration methods for linear signal measurements and linear dependence of the response on th…
Mixup technique improved, reducing manifold mismatch for better calibration.
problem Improving calibration of models using Mixup.
method Dynamic adjustment of interpolation coefficients based on sample similarity.
result Improved predictive performance and calibration with reduced manifold mismatch.
Discovery of an accurate causal Bayesian network structure from observational data can be useful in many areas of science. Often the discoveries are made under uncertainty, which can be expressed as probabilities. To guide the use of such discoveries, including directing further investigation, it is important that thos…
The paper improves Bayesian optimization by calibrating uncertainty estimates.
problem Improper uncertainty estimates in Bayesian optimization when data is non-stationary.
method Proposes online learning algorithms to maintain calibration on non-i.i.d. data and integrates them into Bayesian optimization.
result Calibrated Bayesian optimization converges to better optima in fewer steps.
The paper reviews and extends calibration concepts for classification and regression.
problem Formalizing compatibility between probabilistic predictions and outcomes.
method Review and extension of existing calibration concepts, introduction of new concepts.
result Hierarchical relations between calibration concepts for various data types.
Proposes isotonic regression for calibrating Deep Cox models' survival probabilities.
problem Poor calibration of Deep Cox models' survival probabilities.
method Isotonic regression for post hoc calibration of Deep Cox models.
result Establishes favorable theoretical guarantees and demonstrates empirical effectiveness.
Score calibration enables automatic speaker recognizers to make cost-effective accept / reject decisions. Traditional calibration requires supervised data, which is an expensive resource. We propose a 2-component GMM for unsupervised calibration and demonstrate good performance relative to a supervised baseline on NIST…
New loss improves DNN calibration without sacrificing accuracy.
problem Overfitting and overconfident predictions in DNNs.
method Proposes a new loss function inspired by Bayes decision theory.
result Improves DNN calibration without compromising accuracy.
MCNet improves uncertainty calibration in online advertising by modeling complex relations and balancing performance.
problem Lack of effective calibration for complex relations and context features in online advertising.
method Introduces MCNet with MCF, order-preserving, and field-balance regularizers.
result Superior performance in generating well-calibrated probability predictions on public and industrial datasets.
Proposes top-label calibration and M2B framework for multiclass to binary calibration.
problem Multiclass calibration and interpretation issues.
method Top-label calibration and M2B reduction framework.
result M2B + HB achieves lower calibration error than other methods.
CAMul forecasts with calibrated and accurate multi-view time-series data.
problem Combining diverse data sources for reliable time-series forecasting.
method CAMul integrates multi-modal data views dynamically, assigning importance based on context.
result CAMul outperforms state-of-the-art models by 25% in accuracy and calibration.
Proposes a method to calibrate data for more accurate linear correlation testing.
problem Inaccurate Pearson's correlation coefficient due to sample size and data non-normality.
method Predictive data calibration using machine learning to condition data on expected linear relationship.
result Calibrated Pearson's correlation coefficient yields a calibrated p-value and r estimate for posterior probability interpretation.
New method calibrates uncertainty in molecular property predictions.
problem Uncalibrated uncertainty estimates in molecular property predictions.
method Message Passing Neural Networks with calibrated probabilistic predictive distribution.
result Accurate molecular formation energy predictions with well-calibrated uncertainty.
Adaptive calibration improves model accuracy with fewer simulations.
problem Inefficient calibration of complex models using fixed designs.
method Bayesian adaptive experimental design to optimize simulation runs.
result The method achieves better parameter estimation with fewer simulations.
THERMOMETER calibrates LLMs efficiently for diverse tasks.
problem Calibrating large language models is challenging due to computational and versatility issues.
method THERMOMETER learns an auxiliary model for calibrating a LLM using data from multiple tasks.
result THERMOMETER produces better-calibrated responses for new tasks.
In order to cope with the increased data volumes generated by modern radio interferometers such as LOFAR (Low Frequency Array) or SKA (Square Kilometre Array), fast and efficient calibration algorithms are essential. Traditional radio interferometric calibration is performed using nonlinear optimization techniques such…
Efficiently calibrates SABR/LIBOR models to real market caplets and swaptions data.
problem Calibration of stochastic volatility models to real market data.
method Proposes a parallelized simulated annealing algorithm for multi-GPUs.
result Numerical results show advantages of using multi-GPUs for SABR/LIBOR model calibration.
Schervish (1985b) showed that every forecasting system is noncalibrated for uncountably many data sequences that it might see. This result is strengthened here: from a topological point of view, failure of calibration is typical and calibration rare. Meanwhile, Bayesian forecasters are certain that they are calibrated-…
In many applications, accurate class probability estimates are required, but many types of models produce poor quality probability estimates despite achieving acceptable classification accuracy. Even though probability calibration has been a hot topic of research in recent times, the majority of this has investigated n…
Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
Trained DNN models are increasingly adopted as integral parts of software systems, but they often perform deficiently in the field. A particularly damaging problem is that DNN models often give false predictions with high confidence, due to the unavoidable slight divergences between operation data and training data. To…
The paper calibrates SPX and VIX options using optimal transport.
problem Joint calibration of SPX and VIX options or futures.
method Semimartingale optimal transport problem with PDE formulation and dual formulation.
result The model accurately calibrates SPX, VIX options, and futures simultaneously.
New framework calibrates computer models using deep learning and quantile regression.
problem Uncertainty in computer model input parameters due to high-dimensional time series data.
method Deep neural network with long-short term memory layers for inverse modeling, quantile regression for interval predictions.
result Accurate point and interval estimates for input parameters in WRF-hydro model.
Improves robustness of propensity score estimators in challenging settings.
problem Limited overlap, small sample sizes, or unbalanced data.
method Extends calibration techniques for propensity score models, focusing on sample-splitting schemes.
result Calibration reduces variance and bias in inverse probability weighting and double/debiased machine learning frameworks.
Calibrated ensembles improve both ID and OOD accuracy in distribution shift.
problem Desired balance between in-distribution and out-of-distribution accuracy.
method Ensemble standard and robust models, calibrating on ID data only.
result ID-calibrated ensembles outperform state-of-the-art methods on multiple datasets.
New framework ensures valid uncertainty estimates for any data stream changes.
problem Challenges of distribution shifts and adversarial actors in real-world data streams.
method Leveraging Blackwell approachability from game theory, the framework guarantees calibrated uncertainties for any compact space.
result Improves calibration and decision-making for energy systems.