Unified approach for multicalibration in weakly supervised learning.
problem Existing multicalibration methods require clean input-label pairs, which are unavailable in weakly supervised learning.
method Developed estimators and post-hoc correction methods for multicalibration under weak supervision.
result Unified framework for estimating and correcting multicalibration under weak supervision with finite-sample guarantees.
Two approaches improve conformal Bayes for label shift, one post-hoc and one in-training.
problem Improving prediction sets for target domain under label shift.
method Two complementary approaches: post-hoc calibration and in-training adaptation.
result In-training adaptation achieves up to 43% width reduction at unchanged coverage.
Conformal Bayes under label shift: post-hoc calibration vs. in-training adaptation
problem Bayesian prediction sets under label shift
method Post-hoc calibration vs. In-training adaptation
result Both strategies achieve valid coverage equally in an unbiased training regime
This work proves L2-regularized ERM controls smCE without post-hoc correction.
problem Calibration of predicted probabilities in machine learning models.
method Canonical L2-regularized empirical risk minimization. result Theoretical proof that smCE is controlled by ERM without post-hoc correction.
Framework evaluates post-hoc interpretability methods in time-series classification.
problem Lack of suitable post-hoc interpretability methods for time-series classification.
method Proposes a framework with quantitative metrics to assess interpretability methods.
result Addresses several drawbacks of existing methods, including dependence on human judgement and data distribution shift.
CRC improves multivariate forecasting accuracy without risking performance degradation.
problem Systematic errors and lack of guarantees in multivariate forecasters.
method CRC uses a causality-inspired encoder and hybrid corrector with a safety mechanism.
result CRC consistently improves accuracy and ensures high non-degradation rates.
E-scores assess LLM outputs for correctness, addressing p-hacking issues.
problem Limited principled mechanisms to assess generative model correctness.
method Use e-values to complement LLM outputs with e-scores, providing flexibility in tolerance levels.
result Achieves guarantees of correctness assessment and upper bounds size distortion.
New method calibrates probabilistic regression models without restrictive assumptions.
problem Ensuring predictive distributions accurately reflect true uncertainty.
method Nonparametric re-calibration algorithm based on conditional kernel mean embeddings.
result Consistently outperforms prior re-calibration approaches across various benchmarks.
New method improves model calibration by adjusting confidence based on prediction correctness.
problem Improving model confidence alignment with true class probabilities.
method Post-hoc calibration objective using transformed samples for training.
result Competitive calibration performance on in-distribution and out-of-distribution test sets.
Geometric method improves uncertainty estimation in real-time.
problem Improving uncertainty estimation in machine learning models.
method Geometric distance from training inputs for uncertainty estimation, post-hoc calibration.
result Method yields better uncertainty estimations than existing approaches.
Prototype model improves model auditing and understanding.
problem Auditing and understanding modern language models is expensive and approximate.
method Introduced a sparse, non-negative mixture of learned prototypes trained with clustering objectives.
result Prototype models either surpass or remain within 2.5 percentage points of dense baselines on downstream tasks.
CCAC calibrates DNN classifiers on OOD datasets by separating mis-classified samples.
problem Calibrating DNN classifiers on out-of-distribution datasets is challenging.
method CCAC introduces an auxiliary class to map DNN output to calibrated confidence, separating mis-classified from correctly classified samples.
result CCAC consistently outperforms prior methods on various DNN models, datasets, and applications.
Post-hoc transforms can reverse model performance trends, especially in noisy settings.
problem Post-hoc transforms can reverse model performance trends, especially in noisy settings.
method Empirical study and analysis of post-hoc transforms like temperature scaling, ensembling, and SWA.
result Post-hoc reversal can prevent double descent and mitigate mismatches between test loss and test error.
The paper corrects Bayesian neural network approximations to improve decision quality.
problem Inaccurate posterior approximations in Bayesian neural networks lead to suboptimal decisions.
method Develops methods to calibrate approximate posterior predictive distributions for better decision making.
result Empirically produces higher quality decisions compared to previous methods.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Mathematical study shows post-hoc explanations are better than attention weights alone.
problem Understanding the internal behavior of attention-based models.
method Mathematical analysis of a simple attention-based architecture.
result Post-hoc explanations provide more useful insights than attention weights alone.
CalArena benchmarks post-hoc calibration methods across various tasks.
problem Inconsistent evaluations of post-hoc calibration methods.
method Large-scale benchmark with 2000 experiments, covering diverse models and settings.
result Smooth calibration functions outperform binning-based approaches.
Reweighting improves GAN accuracy without sacrificing statistical power.
problem Improving the fidelity of generative models.
method Post-hoc reweighting function applied to generated examples.
result Weighted GAN examples significantly improve accuracy.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
Study post-hoc Learning to Defer using density-ratio losses.
problem Optimizing decision-making between models and experts.
method Density-ratio losses for post-hoc L2D scorers, derived from class-probability estimation.
result The approach recovers known results and introduces new connections to expert comparison and anomaly detection.
Post-hoc calibration of neural networks using g-Layers proves theoretical justification.
problem Ensuring the confidence of neural network decisions in real-world applications.
method Proves theoretical justification for post-hoc calibration methods by adding g-Layers and minimizing NLL.
result Proves that adding g-Layers and minimizing NLL can lead to a calibrated network.
End-to-end method improves neural network calibration during training.
problem Improving neural network calibration for regression problems.
method Quantile Recalibration Training integrates post-hoc calibration into model training.
result Improved predictive accuracy and calibration in a large-scale experiment.
A post-hoc framework improves model performance by calibrating different feature spaces.
problem Improving AUC performance on binary classification tasks for overconfident models.
method Identifies heterogeneous partitions of the feature space and applies post-hoc calibration techniques to each partition.
result Theoretical optimality of the framework for any model, demonstrated on deep neural networks.
Counterfactual post-hoc interpretability approaches have been proven to be useful tools to generate explanations for the predictions of a trained blackbox classifier. However, the assumptions they make about the data and the classifier make them unreliable in many contexts. In this paper, we discuss three desirable pro…
New methods reduce bias in machine learning predictions for causal inference without extra data.
problem Machine learning predictions from satellite data shrink toward the mean, leading to biased causal estimates.
method Two post-hoc correction methods: Linear Calibration Correction (LCC) and Tweedie's approach, reduce shrinkage-induced bias.
result Tweedie's method yields nearly unbiased treatment-effect estimates, enabling multiple trials with a single map.
Stein importance sampling is a widely applicable technique based on kernelized Stein discrepancy, which corrects the output of approximate sampling algorithms by reweighting the empirical distribution of the samples. A general analysis of this technique is conducted for the previously unconsidered setting where samples…
While statistics and machine learning offers numerous methods for ensuring generalization, these methods often fail in the presence of adaptivity---the common practice in which the choice of analysis depends on previous interactions with the same dataset. A recent line of work has introduced powerful, general purpose a…
New method calibrates deep networks by preserving top-k predictions.
problem Calibrated confidence scores for multi-class deep networks to avoid rare mistakes.
method Intra order-preserving functions combined with neural network architecture.
result Outperforms state-of-the-art methods in evaluation metrics.
Post-hoc explanations improve CNNs by replacing final linear layer with k-means classifier.
problem CNNs lack accurate data representation in their built-in prototypes.
method Introduces k-means-based post-hoc explanations for CNNs, leveraging spatial consistency of convolutional receptive fields.
result Using shallower, less compressed feature activations improves semantic fidelity at the cost of slight predictive performance.
A novel post-hoc calibration method reduces neural network calibration errors.
problem Neural networks produce poorly calibrated probabilities, leading to underconfidence and overconfidence.
method Probability bounding (PB) via box-constrained softmax (BCSoftmax) function.
result Consistently reduces calibration errors on four real-world datasets.
Post-ADC inference corrects bias in statistical inference after active data collection.
problem Bias in inference after active data collection.
method Post-ADC inference framework that corrects bias from both ADC process and data-driven target construction.
result Valid inference for data collected by SMBO methods like GP-UCB and TPE.
Study shows disentanglement models learn correlations from data, impacting fairness.
problem Disentanglement models learn correlations in real-world data, affecting downstream applications.
method Empirical study on 4260 models, analyzing correlations in latent representations.
result Systematically induced correlations are learned by disentanglement models, impacting fairness.
TaylorPODA uses Taylor expansions to improve feature attributions for opaque models.
problem Lack of systematic framework for quantifying feature contributions in opaque models.
method Taylor expansion framework with postulates (precision, federation, zero-discrepancy, adaptation).
result TaylorPODA achieves competitive results and provides principled explanations.
Post-hoc model-agnostic interpretation methods such as partial dependence plots can be employed to interpret complex machine learning models. While these interpretation methods can be applied regardless of model complexity, they can produce misleading and verbose results if the model is too complex, especially w.r.t. f…
Post-detection analysis identifies responsible coordinates for multivariate change-points.
problem Identifying which coordinates in multivariate time series change after a detected change-point.
method Two-sample testing procedures with nonparametric tests for Type I error control.
result Strong performance of proposed post hoc statistical procedures.
New bin-wise scaling methods improve prediction uncertainty calibration for machine learning.
problem Improving prediction uncertainty calibration for machine learning regression.
method Adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and feature-based binning.
result Improved adaptivity and consistency in prediction uncertainty calibration.
Novel flows generate molecules without post-processing.
problem Generating new molecules efficiently and without post-processing issues.
method Continuous normalizing E(3)-equivariant flows based on node ODEs coupled as a graph PDE.
result Generated samples achieve state-of-the-art performance on QM9 and ZINC250K benchmarks.
EAGLE improves reproducibility and stability of model explanations.
problem Creating reliable explanations for opaque machine learning models.
method Formulates perturbation selection as an information-theoretic active learning problem.
result EAGLE learns a linear surrogate model with feature importance scores and uncertainty estimates.
New asymptotic e-values improve inference by eliminating data-dependent scaling inefficiency.
problem Data-dependent scaling inefficiency in existing asymptotic e-values.
method Drawing on Bentkus's near-optimal concentration inequalities, introduce Bentkus-type asymptotic e-values.
result Bentkus-type asymptotic e-values consistently deliver sharper inference than existing alternatives.
Post-hoc uncertainty quantification improves on pre-trained neural networks without underfitting.
problem Uncertainty quantification in neural networks is underfitting or computationally demanding.
method Gaussian Process Activation function (GAPA) for neuron-level uncertainty, with two methods: GAPA-Free and GAPA-Variational.
result GAPA-Variational outperforms Laplace approximation on most datasets in uncertainty quantification metrics.
Improved probabilistic solar irradiance forecasting models for grid integration.
problem Enhancing accuracy of solar irradiance forecasts for grid integration.
method Developed and calibrated probabilistic models using post-hoc calibration techniques.
result NGBoost model with CRUDE calibration achieves comparable performance to numerical weather prediction models.
We describe a post hoc test for the Sharpe ratio, analogous to Tukey's test for pairwise equality of means. The test can be applied after rejection of the hypothesis that all population Signal-Noise ratios are equal. The test is applicable under a simple correlation structure among asset returns. Simulations indicate t…
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
Combines Laplace approximations of deep networks for better uncertainty quantification.
problem Overconfident predictions on outliers in deep learning models.
method Gaussian mixture model posterior using weighted sum of Laplace approximations of pre-trained deep networks.
result Mitigates overconfidence 'far away' from training data.
Enhances deep neural networks with fixed-mean Gaussian processes for uncertainty estimation.
problem Post-hoc uncertainty estimation of pre-trained deep neural networks.
method Fixed-mean Gaussian processes with variational inference for efficient stochastic optimization.
result FMGP improves uncertainty estimation and computational efficiency compared to state-of-the-art methods.
The paper explores how model complexity affects OOD detection performance.
problem Ensuring reliability and safety of machine learning systems through OOD detection.
method Investigates the relationship between model capacity and OOD detection performance using empirical and theoretical analysis.
result The Double Descent phenomenon is observed in post-hoc OOD detection, indicating that overparameterization can enhance OOD detection.
Paper proposes a method to reduce hallucinations in diffusion models using Laplacian score sharpening.
problem Hallucinations in diffusion models create incoherent or unrealistic samples.
method Post-hoc adjustment to the score function during inference using Laplacian approximation.
result Significantly reduces the rate of hallucinated samples across various data types.
New sampling strategy preserves relationships in multivariate scientific data.
problem Reducing storage and enabling efficient multivariate analyses on large scientific data.
method Uses principal component analysis for multivariate data and combines with existing univariate sampling algorithms.
result Efficacy demonstrated on real-world data sets, showing data reduction and multivariate analysis ease.