Post hoc test for Sharpe ratio improves pairwise comparisons.
problem Improving pairwise comparisons of Sharpe ratios.
method Analogous to Tukey's test, applied after rejecting equal Signal-Noise ratios.
result Maintains nominal type I rate and is moderately powerful.
Post-hoc transforms can reverse model performance trends, especially in noisy settings.
problem Post-hoc transforms can reverse model performance trends, especially in noisy settings.
method Empirical study and analysis of post-hoc transforms like temperature scaling, ensembling, and SWA.
result Post-hoc reversal can prevent double descent and mitigate mismatches between test loss and test error.
Post-detection analysis identifies responsible coordinates for multivariate change-points.
problem Identifying which coordinates in multivariate time series change after a detected change-point.
method Two-sample testing procedures with nonparametric tests for Type I error control.
result Strong performance of proposed post hoc statistical procedures.
A post-hoc framework improves model performance by calibrating different feature spaces.
problem Improving AUC performance on binary classification tasks for overconfident models.
method Identifies heterogeneous partitions of the feature space and applies post-hoc calibration techniques to each partition.
result Theoretical optimality of the framework for any model, demonstrated on deep neural networks.
New asymptotic e-values improve inference by eliminating data-dependent scaling inefficiency.
problem Data-dependent scaling inefficiency in existing asymptotic e-values.
method Drawing on Bentkus's near-optimal concentration inequalities, introduce Bentkus-type asymptotic e-values.
result Bentkus-type asymptotic e-values consistently deliver sharper inference than existing alternatives.
The paper tackles confidence calibration for exploratory machine learning problems.
problem Difficulty in curating datasets and confusion about category validity.
method Introduces four new algorithms for category-specific confidence estimation, including kernel density ratios.
result Kernel density ratios provide a novel approach to confidence calibration, especially for exploratory problems.
PH-CS selects test inputs with reliability guarantees, adapting FDR to data.
problem Fixed FDR limits adaptability to downstream needs.
method Post-hoc conformal selection with e-variables, e-BH procedure.
result PH-CS provides reliable FDP estimates and competitive FDR control.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Framework evaluates post-hoc interpretability methods in time-series classification.
problem Lack of suitable post-hoc interpretability methods for time-series classification.
method Proposes a framework with quantitative metrics to assess interpretability methods.
result Addresses several drawbacks of existing methods, including dependence on human judgement and data distribution shift.
Mathematical study shows post-hoc explanations are better than attention weights alone.
problem Understanding the internal behavior of attention-based models.
method Mathematical analysis of a simple attention-based architecture.
result Post-hoc explanations provide more useful insights than attention weights alone.
Unified approach for multicalibration in weakly supervised learning.
problem Existing multicalibration methods require clean input-label pairs, which are unavailable in weakly supervised learning.
method Developed estimators and post-hoc correction methods for multicalibration under weak supervision.
result Unified framework for estimating and correcting multicalibration under weak supervision with finite-sample guarantees.
CalArena benchmarks post-hoc calibration methods across various tasks.
problem Inconsistent evaluations of post-hoc calibration methods.
method Large-scale benchmark with 2000 experiments, covering diverse models and settings.
result Smooth calibration functions outperform binning-based approaches.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
Study post-hoc Learning to Defer using density-ratio losses.
problem Optimizing decision-making between models and experts.
method Density-ratio losses for post-hoc L2D scorers, derived from class-probability estimation.
result The approach recovers known results and introduces new connections to expert comparison and anomaly detection.
While statistics and machine learning offers numerous methods for ensuring generalization, these methods often fail in the presence of adaptivity---the common practice in which the choice of analysis depends on previous interactions with the same dataset. A recent line of work has introduced powerful, general purpose a…
Post-hoc calibration of neural networks using g-Layers proves theoretical justification.
problem Ensuring the confidence of neural network decisions in real-world applications.
method Proves theoretical justification for post-hoc calibration methods by adding g-Layers and minimizing NLL.
result Proves that adding g-Layers and minimizing NLL can lead to a calibrated network.
End-to-end method improves neural network calibration during training.
problem Improving neural network calibration for regression problems.
method Quantile Recalibration Training integrates post-hoc calibration into model training.
result Improved predictive accuracy and calibration in a large-scale experiment.
Counterfactual post-hoc interpretability approaches have been proven to be useful tools to generate explanations for the predictions of a trained blackbox classifier. However, the assumptions they make about the data and the classifier make them unreliable in many contexts. In this paper, we discuss three desirable pro…
The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. …
Bayes-TrEx finds in-distribution examples for model inspection.
problem Challenges in interpreting neural networks, especially high-confidence failures and ambiguous classifications.
method Bayesian sampling approach to find in-distribution examples with specified prediction confidence.
result Bayes-TrEx enables more flexible holistic model analysis than just inspecting the test set.
The paper proposes a new evaluation framework for causal inference models.
problem Challenges in estimating causal effects from observational data.
method Complements evaluation of causal inference models with statistical evidence and non-parametric tests.
result Eliminates the influence of a few instances or simulations on benchmarking results.
S-LIME stabilizes LIME for more reliable model explanations.
problem Instability of post hoc explanation methods like LIME.
method Uses hypothesis testing based on central limit theorem to stabilize explanations.
result Demonstrates effectiveness of S-LIME on simulated and real-world data.
New method calibrates deep networks by preserving top-k predictions.
problem Calibrated confidence scores for multi-class deep networks to avoid rare mistakes.
method Intra order-preserving functions combined with neural network architecture.
result Outperforms state-of-the-art methods in evaluation metrics.
Post-hoc explanations improve CNNs by replacing final linear layer with k-means classifier.
problem CNNs lack accurate data representation in their built-in prototypes.
method Introduces k-means-based post-hoc explanations for CNNs, leveraging spatial consistency of convolutional receptive fields.
result Using shallower, less compressed feature activations improves semantic fidelity at the cost of slight predictive performance.
A novel post-hoc calibration method reduces neural network calibration errors.
problem Neural networks produce poorly calibrated probabilities, leading to underconfidence and overconfidence.
method Probability bounding (PB) via box-constrained softmax (BCSoftmax) function.
result Consistently reduces calibration errors on four real-world datasets.
FCDD improves image anomaly detection without post-hoc explainers.
problem Image anomaly detection, especially pixel-wise.
method Fully Convolutional Data Description (FCDD) directly addresses anomaly detection without post-hoc methods.
result FCDD achieves state-of-the-art results on pixel-wise AD tasks.
TaylorPODA uses Taylor expansions to improve feature attributions for opaque models.
problem Lack of systematic framework for quantifying feature contributions in opaque models.
method Taylor expansion framework with postulates (precision, federation, zero-discrepancy, adaptation).
result TaylorPODA achieves competitive results and provides principled explanations.
TULiP estimates uncertainty for deep learning models safely.
problem Reliable uncertainty estimation for deep learning models in the open world.
method TULiP considers a hypothetical perturbation, bounds its effect, and computes uncertainty from sampled predictions.
result TULiP achieves state-of-the-art performance in OOD detection benchmarks.
Post-hoc model-agnostic interpretation methods such as partial dependence plots can be employed to interpret complex machine learning models. While these interpretation methods can be applied regardless of model complexity, they can produce misleading and verbose results if the model is too complex, especially w.r.t. f…
Conformal Bayes under label shift: post-hoc calibration vs. in-training adaptation
problem Bayesian prediction sets under label shift
method Post-hoc calibration vs. In-training adaptation
result Both strategies achieve valid coverage equally in an unbiased training regime
Study compares various calibration methods for binary classification tasks.
problem Improving probabilistic predictions in binary classification models.
method Benchmarked 21 classifiers using 5 calibration methods on real data.
result Venn-Abers predictors and Beta calibration show the largest log-loss reductions.
EAGLE improves reproducibility and stability of model explanations.
problem Creating reliable explanations for opaque machine learning models.
method Formulates perturbation selection as an information-theoretic active learning problem.
result EAGLE learns a linear surrogate model with feature importance scores and uncertainty estimates.
Post-hoc uncertainty quantification improves on pre-trained neural networks without underfitting.
problem Uncertainty quantification in neural networks is underfitting or computationally demanding.
method Gaussian Process Activation function (GAPA) for neuron-level uncertainty, with two methods: GAPA-Free and GAPA-Variational.
result GAPA-Variational outperforms Laplace approximation on most datasets in uncertainty quantification metrics.
This paper combines existing OOD detection methods to improve overall performance.
problem Improving robustness of neural networks in safety-critical applications.
method Integrates four strategies for combining multiple OOD detection scores.
result Enhanced OOD detection through multi-dimensional evaluation metrics.
A new OOD detection method OTOD uses optimal transport theory to improve model performance.
problem Detecting unknown samples in real-world machine learning models.
method OTOD uses optimal transport theory to calculate an OOD score combining features, logits, and softmax probability space.
result OTOD outperforms state-of-the-art methods by significant margins on benchmarks.
This work approximates full conformal prediction for neural networks without sample splitting.
problem Uncertainty quantification for neural network regression models.
method Approximating full conformal prediction using Gauss-Newton influence for post-hoc uncertainty estimation.
result Locally-adaptive and often tighter prediction intervals compared to split-CP.
Improved probabilistic solar irradiance forecasting models for grid integration.
problem Enhancing accuracy of solar irradiance forecasts for grid integration.
method Developed and calibrated probabilistic models using post-hoc calibration techniques.
result NGBoost model with CRUDE calibration achieves comparable performance to numerical weather prediction models.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
Paper shows LIME and SHAP explanations can be fooled by adversarial attacks.
problem Reliability of post hoc explanation methods like LIME and SHAP.
method Adversarial attack technique to hide classifier biases.
result Popular explanation techniques can be fooled into generating biased explanations.
Two approaches improve conformal Bayes for label shift, one post-hoc and one in-training.
problem Improving prediction sets for target domain under label shift.
method Two complementary approaches: post-hoc calibration and in-training adaptation.
result In-training adaptation achieves up to 43% width reduction at unchanged coverage.
Combines Laplace approximations of deep networks for better uncertainty quantification.
problem Overconfident predictions on outliers in deep learning models.
method Gaussian mixture model posterior using weighted sum of Laplace approximations of pre-trained deep networks.
result Mitigates overconfidence 'far away' from training data.
FLANs process each feature separately for better interpretability.
problem Need for interpretable machine learning models in critical scenarios.
method Feature-wise latent representations summed for prediction.
result FLANs enhance interpretability without sacrificing performance.
New method improves model calibration by adjusting confidence based on prediction correctness.
problem Improving model confidence alignment with true class probabilities.
method Post-hoc calibration objective using transformed samples for training.
result Competitive calibration performance on in-distribution and out-of-distribution test sets.
GAPA method provides efficient uncertainty quantification for pretrained networks.
problem Reliable uncertainty estimates for pretrained models are challenging.
method Post-hoc Gaussian Process Activations (GAPA) method that shifts Bayesian modeling from weights to activations.
result GAPA method provides efficient uncertainty quantification without altering the backbone's predictions.
Enhances deep neural networks with fixed-mean Gaussian processes for uncertainty estimation.
problem Post-hoc uncertainty estimation of pre-trained deep neural networks.
method Fixed-mean Gaussian processes with variational inference for efficient stochastic optimization.
result FMGP improves uncertainty estimation and computational efficiency compared to state-of-the-art methods.
The paper explores how model complexity affects OOD detection performance.
problem Ensuring reliability and safety of machine learning systems through OOD detection.
method Investigates the relationship between model capacity and OOD detection performance using empirical and theoretical analysis.
result The Double Descent phenomenon is observed in post-hoc OOD detection, indicating that overparameterization can enhance OOD detection.
New sampling strategy preserves relationships in multivariate scientific data.
problem Reducing storage and enabling efficient multivariate analyses on large scientific data.
method Uses principal component analysis for multivariate data and combines with existing univariate sampling algorithms.
result Efficacy demonstrated on real-world data sets, showing data reduction and multivariate analysis ease.
Unified calibration metrics improve forecast sharpness and accuracy.
problem Improving the sharpness of probabilistic forecasts while maintaining calibration.
method Kernel-based calibration metrics that unify and generalize existing methods for classification and regression.
result Enhanced calibration, sharpness, and decision-making across various tasks.