Comparison of decision curve analysis and cost curves for model evaluation.
problem Evaluating classification performance across different operating contexts.
method Comparison of Decision Curve Analysis (DCA) and Cost Curves.
result DCA and Cost Curves are closely related, with Brier curves being more generally applicable.
Calibrating classifiers reduces grouping loss using sufficiency criteria.
problem Grouping loss in probabilistic classifier calibration is often overlooked.
method Revisited Langford & Zadrozny's probing reduction approach and introduced Brier curves.
result The probing reduction approach reduces grouping loss and supports sufficient calibration.
Critiques binary classification evaluation methods, advocating for proper scoring rules.
problem The dominance of top-K metrics and fixed-threshold evaluations in machine learning.
method Introduces a decision-theoretic framework mapping evaluation metrics to their use cases, and implements a clipped Brier score variant.
result Demonstrates the clinical utility of proper scoring rules through a Python package, exttt{briertools}.
Develops regression trees for estimating cumulative incidence curves in competing risks.
problem Estimating cumulative incidence functions in competing risks settings.
method Uses augmented estimators of the Brier score risk to build and prune regression trees.
result Demonstrates the utility of the proposed methods through simulation studies and real data.
The Brier score is commonly used for evaluating probability predictions. In survival analysis, with right-censored observations of the event times, this score can be weighted by the inverse probability of censoring (IPCW) to retain its original interpretation. It is common practice to estimate the censoring distributio…
Reduces quantifier variance with accuracy optimization of base classifier.
problem Minimizing quantifier variance under prior probability shift.
method Optimizes the Brier score of a base classifier for training data.
result Optimizing Brier score on training data reduces quantifier variance on test data.
A new method calibrates forecasts without sacrificing expertise.
problem Forecasters' calibration scores can be manipulated to appear expert.
method Deterministic and stochastic online procedures to calibrate forecasts.
result Calibration can be achieved without losing expertise.
Certified calibration methods protect model confidence from adversarial attacks.
problem Adversarial attacks degrade model calibration, reducing confidence in predictions.
method Developed certified calibration methods to provide worst-case bounds on calibration under adversarial perturbations.
result Certified calibration methods produce analytic and approximate bounds for the Brier score and expected calibration error.
Proposes and evaluates three diagnostic graphics for probabilistic classifiers.
problem Evaluating and comparing probabilistic classifiers.
method Triptych of diagnostic graphics: reliability diagram, ROC curve, Murphy diagram.
result Visual diagnostics reveal distinct aspects of forecast performance.
Foresight Arena benchmarks AI forecasting on real-world markets, isolating predictive edge.
problem Evaluating AI forecasting ability in real-world markets is challenging due to overfitting, centralized trust, and conflated metrics.
method Permissionless, on-chain benchmark using probabilistic forecasts, commit-reveal protocol, and smart contracts.
result Demonstrates the need for 350 predictions to reliably distinguish agents of different skill levels.
Calibrated models can lead to miscalibrated aggregations in strategic interactions.
problem Miscalibration in aggregated predictions from multiple calibrated models.
method Analysis of strategic interactions between calibrated predictors, proving conditions for miscalibration and comparing VCG and Brier-score aggregation methods.
result VCG aggregation method outperforms Brier-score in strategic settings, providing robustness and comparable accuracy.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
BYOV combines SSL and Bayesian methods for uncertainty estimation.
problem Model uncertainty in applications.
method Combines Bootstrap Your Own Latent (BYOL) and Bayes by Backprop (BBB).
result BYOV improves model calibration and reliability with various augmentations.
This paper enhances the Random Survival Forest model for better predictive maintenance.
problem Improving time-to-failure estimation in predictive maintenance with Random Survival Forest.
method A three-level framework for quantifying hyperparameter tunability, including model-level and hyperparameter-level metrics.
result Hyperparameter tuning significantly improves Random Survival Forest model performance in predictive maintenance.
A new framework separates classifier calibration and discrimination.
problem Combining reliability and resolution in probabilistic predictions.
method Manokhin Probability Matrix separates reliability and resolution using Spiegelhalter Z-statistic and AUC-ROC.
result Classifiers are categorized into four archetypes: Eagle, Bull, Sloth, and Mole.
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
New metric reduces estimation error in survival model evaluation.
problem Dependent censoring complicates survival model evaluation.
method Dependent Brier score based on Archimedean copula and Copula-Graphic estimator.
result Reduces estimation error by 12-16% on average.
The paper breaks down AUC into cluster-level components for better model diagnostics.
problem Global AUC masks weaknesses in specific subpopulations, leading to financial or operational risks.
method Formal decomposition of AUC into intra- and inter-cluster components, comparing with other performance metrics.
result Allows practitioners to evaluate and diagnose model performance within and across clusters.
CNNs improve wind speed forecasts in the Netherlands.
problem Limited spatial patterns in current post-processing methods.
method Convolutional Neural Networks (CNNs) for spatial wind speed information.
result CNNs produce better probabilistic forecasts with higher Brier skill scores.
Survival MDN uses invertible functions to speed up survival analysis models.
problem Training neural ODEs for survival analysis is computationally expensive.
method Survival MDN applies an invertible positive function to MDN outputs.
result Survival MDN outperforms or matches other models on concordance, Brier score, and log-likelihood.
A common method of generalizing binary to multi-class classification is the error correcting code (ECC). ECCs may be optimized in a number of ways, for instance by making them orthogonal. Here we test two types of orthogonal ECCs on seven different datasets using three types of binary classifier and compare them with t…
Improves reliability diagrams for probabilistic forecasts.
problem Lack of stability in reliability diagrams hampered their use.
method CORP approach using non-parametric isotonic regression and PAV algorithm.
result Improved reliability diagrams with statistical consistency and reproducibility.
An accurate model of a patient's individual survival distribution can help determine the appropriate treatment for terminal patients. Unfortunately, risk scores (e.g., from Cox Proportional Hazard models) do not provide survival probabilities, single-time probability models (e.g., the Gail model, predicting 5 year prob…
Innovative game theory approach optimizes survival analysis metrics.
problem Survival analysis models trained with maximum likelihood do not directly optimize criteria like Brier score or Bernoulli log likelihood.
method Inverse-Weighted Survival Games: Construct objectives from re-weighted estimates featuring the other model, holding the latter fixed during training.
result Games optimize Brier score on simulations and real-world data.
Survival analysis/time-to-event models are extremely useful as they can help companies predict when a customer will buy a product, churn or default on a loan, and therefore help them improve their ROI. In this paper, we introduce a new method to calculate survival functions using the Multi-Task Logistic Regression (MTL…
Transformer model outperforms classical methods in childhood anemia prediction across diverse countries.
problem Generalizing childhood anemia prediction models across different countries and data scarcity.
method Transformer-based tabular foundation model compared to classical supervised methods using DHS data.
result Transformer model achieves lower Brier score and ECE in low-data settings, outperforming classical models.
Comprisk simplifies competing-risks analysis in Python.
problem Analyzing medical time-to-event data with competing risks.
method A scikit-learn-compatible toolkit for competing-risks survival analysis.
result Comprisk provides a unified API for various competing-risks methods.
New methods for time-to-event prediction are proposed by extending the Cox proportional hazards model with neural networks. Building on methodology from nested case-control studies, we propose a loss function that scales well to large data sets, and enables fitting of both proportional and non-proportional extensions o…
The goal of lifetime clustering is to develop an inductive model that maps subjects into K clusters according to their underlying (unobserved) lifetime distribution. We introduce a neural-network based lifetime clustering model that can find cluster assignments by directly maximizing the divergence between the empiri…
SSH-Net: A Deep Neural Network for Predicting Failure Time Distribution Functions under Competing Risks with GPU Data
problem Predicting failure time distribution functions under competing risks
method Structured Segmented Hazard Deep Neural Network (SSH-Net)
result Prediction accuracy validated through simulation studies and GPU data
Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the exist…
ABC improves uncertainty quantification in LLMs for clinical diagnostics.
problem Overconfident and poorly calibrated estimates of LLMs in clinical domains.
method Approximate Bayesian Computation (ABC) for likelihood-free inference.
result Improves accuracy by up to 46.9%, reduces Brier scores by 74.4%, and enhances calibration.
In this manuscript we analyze a data set containing information on children with Hodgkin Lymphoma (HL) enrolled on a clinical trial. Treatments received and survival status were collected together with other covariates such as demographics and clinical measurements. Our main task is to explore the potential of machine …
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
The study improves theoretical understanding of using multiple synthetic datasets for better model accuracy.
problem Lack of theoretical understanding of using multiple synthetic datasets for supervised learning.
method Derive bias-variance decompositions for multiple synthetic datasets settings.
result A simple rule of thumb to select the appropriate number of synthetic datasets.
In this paper, we propose a novel maximum causal Tsallis entropy (MCTE) framework for imitation learning which can efficiently learn a sparse multi-modal policy distribution from demonstrations. We provide the full mathematical analysis of the proposed framework. First, the optimal solution of an MCTE problem is shown …
Binary classification is highly used in credit scoring in the estimation of probability of default. The validation of such predictive models is based both on rank ability, and also on calibration (i.e. how accurately the probabilities output by the model map to the observed probabilities). In this study we cover the cu…
Simple mode exploration methods do not improve performance in neural networks.
problem Improving predictive probabilities in neural networks.
method Exploring local regions around diverse solutions using simple methods.
result Simple mode exploration methods do not improve performance.
SurvSurf predicts first hitting times for intermittent events without monotonic violations.
problem Predicting first hitting times for intermittent events with monotonicity guarantees.
method Partially monotonic neural network for sequential events, incorporating unobserved events.
result SurvSurf outperforms existing models in MSE and IBS metrics.
A new survival analysis method eliminates hyperparameter tuning.
problem Survival analysis model hyperparameters are difficult to tune.
method Flexible survival density estimation with importance sampling.
result Matches or outperforms existing methods on real-world datasets.
KCal calibrates deep networks by embedding logits in a metric space.
problem Overconfident predictions from DNNs, especially in high-risk applications.
method KCal learns a metric space on the penultimate-layer latent embedding and generates predictions using kernel density estimates.
result KCal provides a provable full calibration guarantee and consistently outperforms baselines.
Study evaluates three class imbalance techniques across diverse datasets.
problem Class imbalance in binary classification tasks.
method Synthetic Minority Over-sampling Technique (SMOTE), Class Weights tuning, Decision Threshold Calibration.
result Decision Threshold Calibration is the most consistently effective technique.
This paper considers fair probabilistic binary classification where the outputs of primary interest are predicted probabilities, commonly referred to as scores. We formulate the problem of transforming scores to satisfy fairness constraints that are linear in conditional means of scores while minimizing a cross-entropy…
The number of trees T in the random forest (RF) algorithm for supervised learning has to be set by the user. It is controversial whether T should simply be set to the largest computationally manageable value or whether a smaller T may in some cases be better. While the principle underlying bagging is that "more trees a…
This work broadens calibeating to various proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses.
method Regret minimization and Bregman divergence approach.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
This work generalizes calibeating for a broader range of proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses beyond Brier and log loss.
method Regret minimization based on Bregman divergence for a family of proper losses.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
A new ensemble method improves kNN performance by extending the neighborhood rule.
problem Traditional kNN's limitations when test points are outside the spherical region and ensemble's high errors.
method Determines neighbors in k steps, using bootstrap samples and optimal models selection.
result The proposed ensemble method outperforms state-of-the-art methods on 17 benchmark datasets.
SAWAR improves SA models by making them more robust to data uncertainties.
problem Improving survival analysis models' robustness to data uncertainties.
method Adversarial robustness through Min-Max optimization with CROWN-IBP.
result SAWAR outperforms baseline methods and SOTA models across various metrics.