Survey of methods to calibrate neural network predictions.
problem Ensuring neural networks provide accurate confidence levels.
method Empirical comparison of calibration methods.
result Various techniques for calibrating neural networks.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “calibration problem”
Unique solutions found for Plateau problems in smooth and continuous calibrations.
problem Finding unique solutions to the Plateau problem for specific types of currents.
method Boundary regularity theory for area-minimizing currents and unique continuation argument.
result Every compactly supported smoothly or continuously calibrated integral current is the unique solution to the Plateau problem for its boundary data.
In many classification problems it is desirable to output well-calibrated probabilities on the different classes. We propose a robust, non-parametric method of calibrating probabilities called SplineCalib that utilizes smoothing splines to determine a calibration function. We demonstrate how applying certain transforma…
Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Errorcs.LG
We propose a new framework to improve the calibration of neural networks.
problem Improving the accuracy of model confidence predictions.
method Introducing a differentiable surrogate for expected calibration error (DECE) and a meta-learning framework to optimise model hyper-parameters for validation set calibration.
result Achieved competitive performance with existing calibration approaches.
Two methods improve calibration of probabilistic classifiers, especially for multi-class problems.
problem Improving calibration of probabilistic classifiers, especially for multi-class problems.
method Two techniques: reduced calibration and class-wise calibration.
result Class-wise reduced calibration algorithms reduce prediction and per-class calibration errors.
In order to cope with the increased data volumes generated by modern radio interferometers such as LOFAR (Low Frequency Array) or SKA (Square Kilometre Array), fast and efficient calibration algorithms are essential. Traditional radio interferometric calibration is performed using nonlinear optimization techniques such…
The paper tackles confidence calibration for exploratory machine learning problems.
problem Difficulty in curating datasets and confusion about category validity.
method Introduces four new algorithms for category-specific confidence estimation, including kernel density ratios.
result Kernel density ratios provide a novel approach to confidence calibration, especially for exploratory problems.
Simple algorithm achieves distance to calibration error of at most 2√T+1.
problem Achieving distance to calibration error of O(√T) in adversarial setting.
method An extremely simple, efficient, deterministic algorithm.
result Obtains distance to calibration error at most 2√T+1.
New algorithm tests model calibration in nearly-linear time.
problem Testing model calibration from samples efficiently.
method Reformulated as minimum-cost flow, solved with dynamic programming.
result Optimal testing problem solved in nearly-linear time.
Survey of Optimal Transport for model calibration.
problem Model calibration using Optimal Transport.
method General framework and numerical algorithms for various models.
result Calibration of volatility models and path-dependent options.
The paper calibrates SPX and VIX options using optimal transport.
problem Joint calibration of SPX and VIX options or futures.
method Semimartingale optimal transport problem with PDE formulation and dual formulation.
result The model accurately calibrates SPX, VIX options, and futures simultaneously.
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.
Nested dichotomies are used as a method of transforming a multiclass classification problem into a series of binary problems. A tree structure is induced that recursively splits the set of classes into subsets, and a binary classification model learns to discriminate between the two subsets of classes at each node. In …
New method calibrates multi-class predictions efficiently without sacrificing accuracy.
problem Efficiently calibrating multi-class predictions without sacrificing accuracy.
method Formulated robust projected smooth calibration and new recalibration algorithms.
result Achieves strong guarantees for binary classification tasks with polynomial complexity.
Study on estimating conditional risk in machine learning.
problem Estimating expected loss of prediction models given input features.
method Analyzed in classification and regression settings, showing equivalence to standard regression. Developed theoretical insights and empirical validation.
result Conditional risk calibration is distinct from existing uncertainty quantification problems.
We study consistency properties of surrogate loss functions for general multiclass learning problems, defined by a general multiclass loss matrix. We extend the notion of classification calibration, which has been studied for binary and multiclass 0-1 classification problems (and for certain other specific learning pro…
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
New algorithm achieves online calibration in polynomial time for high-dimensional problems.
problem Online calibration of high-dimensional probability distributions over many days.
method Randomly selects among sub-forecasters, each predicting empirical outcome frequency over recent time windows.
result Achieves asymptotically calibrated strategies after polynomial number of rounds, resolving open questions.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Generative models often misrepresent class frequencies; this paper calibrates them.
problem Miscalibration of class frequencies in generative models.
method Formulated as constrained optimization, using surrogate objectives to approximate constraints.
result Significant reduction in calibration error across various models and applications.
The paper addresses decision making with partially calibrated forecasts, offering a robust approach.
problem Developing a decision-making strategy for forecasts that are only partially calibrated.
method A minimax approach to mapping predictions to actions, considering worst-case distributions.
result The minimax optimal decision rule is to trust predictions and act accordingly, even for partially calibrated forecasts.
Isotonic regression binning affects calibration statistics of machine learning models.
problem Isotonic regression binning introduces aleatoric uncertainty in calibration statistics.
method Calibration error statistics are recalibrated using isotonic regression, which produces stratified uncertainties.
result Stratified uncertainties lead to significant differences in bin-based calibration statistics.
The task of calibration is to retrospectively adjust the outputs from a machine learning model to provide better probability estimates on the target variable. While calibration has been investigated thoroughly in classification, it has not yet been well-established for regression tasks. This paper considers the problem…
Deep ensembles don't necessarily improve calibration in low data regimes.
problem Calibration issues in deep learning models, especially in low data regimes.
method Examination of data-augmentation, ensembling, and post-processing calibration methods.
result Standard ensembling techniques can lead to less calibrated models in low data regimes.
New method calibrates models under covariate shifts.
problem Calibration of models can be lost under covariate shifts.
method Importance sampling based approach.
result Efficacy demonstrated on real-world and synthetic datasets.
Proposes top-label calibration and M2B framework for multiclass to binary calibration.
problem Multiclass calibration and interpretation issues.
method Top-label calibration and M2B reduction framework.
result M2B + HB achieves lower calibration error than other methods.
Paper proposes a unified framework for evaluating calibration of probabilistic models.
problem Evaluation of calibration for general probabilistic predictive models.
method Unified framework for calibration evaluation and tests for any probabilistic model.
result Generalization and reformulation of existing measures and tests.
We propose nonparametric methods for individual calibration in regression models.
problem Uncertainty quantification and individual calibration for regression models.
method Nonparametric methods agnostic of the underlying model, combining nonparametric and covering number arguments.
result Established matching upper and lower bounds for calibration error.
Proximal algorithms applied to current deformation into cycles.
problem Deformation of de Rham currents into cycles.
method Proximal algorithms, total variation denoising for differential forms.
result Calibrated cycles constructed in calibrated manifolds.
New truthful calibration errors improve model ranking in multiclass prediction.
problem Non-truthful calibration errors can mislead model comparisons.
method Introduced perfectly truthful calibration errors for multiclass predictions.
result Truthful calibration errors preserve decision-theoretic dominance and stabilize model rankings.
The paper presents methods to improve uncertainty calibration in Bayesian Neural Networks.
problem Uncalibrated Bayesian Neural Networks often lead to overconfidence.
method The paper uses alpha-divergences from Information Geometry for calibration.
result Calibration using alpha-divergences provides better uncertainty estimates and is more efficient.
We show that the existence of a computationally efficient calibration algorithm, with a low weak calibration rate, would imply the existence of an efficient algorithm for computing approximate Nash equilibria - thus implying the unlikely conclusion that every problem in PPAD is solvable in polynomial time.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
Optimizes calibration error estimators for better classifier trustworthiness.
problem Lack of guidance on selecting and tuning calibration error estimators.
method Reformulates calibration estimation as a regression problem with i.i.d. input pairs.
result Demonstrates the effectiveness of optimized calibration estimators on image classification tasks.
Validation of ML-UQ calibration statistics using simulated reference values: a sensitivity analysisstat.ML
Study validates ML-UQ calibration statistics using simulated reference values.
problem Validation of ML-UQ calibration statistics is lacking due to lack of predefined reference values.
method Proposed validation workflow using simulated reference values derived from synthetic datasets.
result Some statistics, like CC and ENCE, are overly sensitive to generative distribution choice.
New framework allows selective removal of stale data in option calibration.
problem Inability to remove old data from calibrated option pricing models without full retraining.
method Introduces operator-theoretic Gauss-Newton framework for selective forgetting.
result Provides stability guarantees and perturbation bounds for selective data removal.
This paper argues against using calibration metrics for assessing posterior probabilities and proposes expected proper scoring rules instead.
problem The assessment of posterior probabilities generated by machine learning classifiers using calibration metrics is flawed and should be replaced with expected proper scoring rules.
method The paper reviews proper scoring rules from a practical perspective, explains why expected PSRs are a principled measure of posterior quality, and introduces a new calibration metric called calibration loss.
result Calibration loss is superior to expected calibration error and expected score divergence calibration metrics for assessing posterior probabilities.
Faster online calibration without randomization: interval forecasts and the power of two choicescs.LG
New method achieves faster calibration without randomization.
problem Calibrating probabilistic forecasts in adversarial settings.
method Using interval forecasts and the power of two choices.
result Achieves calibration error rate without randomization.
New method improves calibration of BayesCG for better uncertainty quantification.
problem Bayesian conjugate gradient method's poor calibration limits its utility.
method Randomized postiteration strategy to enhance posterior calibration.
result The method improves the distribution of posterior errors and enhances uncertainty quantification.
Improves model calibration and selection in unsupervised domain adaptation.
problem Distribution shifts in unsupervised domain adaptation.
method Developed a novel importance weighted group accuracy estimator.
result Improves state-of-the-art performances by 22% in model calibration and 14% in model selection.
Signal inference with unknown response: Calibration-uncertainty renormalized estimatorphysics.data-an
The calibration of a measurement device is crucial for every scientific experiment, where a signal has to be inferred from data. We present CURE, the calibration uncertainty renormalized estimator, to reconstruct a signal and simultaneously the instrument's calibration from the same data without knowing the exact calib…
Calibration of Local Volatility Models with Stochastic Interest Rates using Optimal Transportq-fin.MF
Method calibrates stock price models with stochastic interest rates using optimal transport.
problem Calibrating stock price models with stochastic interest rates.
method Non-parametric, semimartingale optimal transport, solving a fully non-linear Hamilton-Jacobi-Bellman equation.
result Fully calibrated model closest to a reference model in a defined cost function.
New method improves calibration of neural networks by targeting robust margins and local smoothness.
problem Poor calibration of neural networks, leading to unreliable confidence estimates.
method Intervene on training procedure by targeting robust margins and local smoothness.
result Improved out-of-sample calibration without sacrificing accuracy.
Study on computing and estimating calibration distance, showing hardness and efficiency.
problem Computing and estimating calibration distance under different assumptions.
method Efficient algorithm for exact computation, polynomial-time approximation scheme; sample-based estimation for upper bounds.
result The problem becomes NP-hard when assumptions are removed, but efficient algorithms exist under certain conditions.
RoPE framework calibrates misspecified simulators for reliable inference.
problem Misspecification compromises reliability of simulation-based inference.
method Data-driven calibration using optimal transport and a small calibration set.
result RoPE framework improves inference accuracy and uncertainty calibration.
Unified framework for generalized Venn and Venn-Abers calibration for reliable prediction.
problem Asymptotic guarantees of popular distribution-free methods in model calibration.
method Unified framework extending Vovk's approach to generic loss functions, transforming predictors into set-valued predictions.
result Finite-sample set predictions shrink to a single conditionally calibrated prediction, capturing epistemic uncertainty.
We extend a variational framework to estimate calibration errors for divergences.
problem Ensuring predicted probabilities match observed class frequencies in machine learning.
method Extend variational framework to divergences, separating over- and under-confidence.
result Avoids overestimation and separates over- and under-confidence.
Study shows heavy-tailed distributions affect reliability of machine learning calibration statistics.
problem Reliability of calibration statistics for machine learning regression tasks is affected by heavy-tailed uncertainty and error distributions.
method Examined two calibration error estimation methods (CE and ZMS) and found ZMS to be less sensitive to heavy-tailed distributions.
result Heavy-tailed distributions make MSE and MV unreliable, but ZMS remains a reliable approach.