CP-ROC bands improve graph classification accuracy and uncertainty quantification.
problem Uncertainty quantification and robustness to distributional shifts in graph classification.
method Conditional Prediction ROC (CP-ROC) bands for graph classification, developed for TGNNs and adaptable to GNNs.
result Statistically guaranteed coverage for CP-ROC under local exchangeability condition, improving prediction reliability.
Rapid overlay of chemical structures (ROCS) is a standard tool for the calculation of 3D shape and chemical ("color") similarity. ROCS uses unweighted sums to combine many aspects of similarity, yielding parameter-free models for virtual screening. In this report, we decompose the ROCS color force field into "color com…
New ROC tools assess predictive abilities for any linearly ordered outcomes.
problem Fundamental restriction in ROC analysis for non-dichotomous outcomes.
method ROC movies and UROC curves for linearly ordered outcomes.
result CPA equals AUC for binary outcomes and relates to Spearman's coefficient for pairwise distinct outcomes.
A scalable ROC-SVM variant reduces training time for imbalanced binary classification.
problem High computational cost of ROC-SVM for imbalanced binary classification.
method Incomplete U-statistics and low-rank kernel approximation.
result Comparable AUC performance with reduced training time.
DCC separates marginal estimation from dependence modeling for improved classification accuracy.
problem Classifying with strong dependence between features.
method Deep Copula Classifier using neural copula densities.
result Achieves excess-risk O(n−r/(2r+d)) for r-smooth copulas. Novel method for multiclass ROC curves using multidimensional Gini index.
problem Multiclass performance evaluation, especially for imbalanced datasets.
method Extends ROC curve methodology to multiclass settings using multidimensional Gini index.
result Validated through case studies in health care and finance.
Receiver operating characteristic (ROC) analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under ROC curve (AUC) with standard cross-validation methods suffers from a large bias. The leave-pair-out (LPO) cross-validation has been shown to correct this bias. Howe…
The fuzzy ROC extends Receiver Operating Curve (ROC) visualization to the situation where some data points, falling in an indeterminacy region, are not classified. It addresses two challenges: definition of sensitivity and specificity bounds under indeterminacy; and visual summarization of the large number of possibili…
The study improves the assessment of fairness in face recognition using ROC curves and statistical guarantees.
problem Improving the assessment of fairness in face recognition systems.
method Proves asymptotic guarantees for empirical ROC curves and fairness metrics, and introduces a recentering technique to avoid bootstrap pitfalls.
result Demonstrates the practical relevance of the methods for assessing fairness in face recognition systems.
The study examines how verifier imperfections impact test-time scaling techniques.
problem Understanding how verifier imperfections affect test-time scaling methods.
method Proves the instance-level accuracy of Best-of-N and Rejection Sampling methods using the geometry of the verifier's ROC curve.
result RS outperforms BoN for fixed compute, but both converge to the same accuracy in the infinite-compute limit.
The Receiver Operating Characteristic (ROC) curve is a representation of the statistical information discovered in binary classification problems and is a key concept in machine learning and data science. This paper studies the statistical properties of ROC curves and its implication on model selection. We analyze the …
Paper estimates optimal ROC curve arc length and AUC, improving classification performance.
problem Estimating optimal ROC curve arc length and AUC in imbalanced binary classification.
method Expresses arc length and AUC as variational objectives, estimating using positive and negative samples.
result Proposed classification procedure maximizes an approximate lower bound of maximal AUC.
Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
Improves ROC/AUC for multi-class classification.
problem Lack of sensible plots, sensitivity to imbalanced data, inability to specify mis-classification cost, and lack of evaluation uncertainty quantification.
method Factorizes multi-class ROC into a one-dimensional vector representation for visualization and summary.
result Provides a binary AUC-equivalent summary and mis-classification weights specification.
Neural network models improve ROC curve evaluation of biomarkers, focusing on age's role in physical activity-mortality association.
problem Improving biomarker evaluation using machine learning for complex relationships.
method Proposes neural network-based covariate-adjusted ROC modeling.
result Age has distinct effects on mortality outcomes when physical activity is measured as total activity time.
New optimization method improves AUC for binary classification and changepoint detection.
problem Non-convex AUC and sub-optimal points in ROC curves.
method AUM (Area Under Min(FP, FN)) surrogate loss function based on sorting and summing ROC curve points.
result AUM minimization learning algorithm improves AUC and speeds up compared to previous methods.
ABROCA assesses algorithmic bias, revealing skewed distributions that inflate results.
problem Detecting nuanced performance differences in classifier fairness.
method Study of ABROCA metric's statistical properties under various conditions.
result ABROCA distributions are skewed, inflating results by chance in imbalanced classes.
Many problems that appear in biomedical decision making, such as diagnosing disease and predicting response to treatment, can be expressed as binary classification problems. The costs of false positives and false negatives vary across application domains and receiver operating characteristic (ROC) curves provide a visu…
Feature selection aims to select the smallest subset of features for a specified level of performance. The optimal achievable classification performance on a feature subset is summarized by its Receiver Operating Curve (ROC). When infinite data is available, the Neyman- Pearson (NP) design procedure provides the most e…
Proposes MCC-F1 curve for better binary classification evaluation.
problem Misleading performance evaluations with ROC and PR curves for imbalanced data.
method Introduces MCC-F1 curve combining MCC and F1 score.
result MCC-F1 curve provides clearer classifier differentiation.
The paper tackles fairness in scoring functions for binary classification.
problem Fairness in scoring functions for binary classification tasks.
method Introduces ROC-based fairness constraints and learning algorithms.
result Generalization bounds and practical learning algorithms for fair scoring functions.
Develops a new method for statistical optimal allocation problems.
problem Statistical optimal allocation problems with constraints.
method Functional differentiability approach and Hadamard differentiability of value functions.
result Validates margin assumption for fast convergence rate of plug-in methods.
Disease classification is a crucial element of biomedical research. Recent studies have demonstrated that machine learning techniques, such as Support Vector Machine (SVM) modeling, produce similar or improved predictive capabilities in comparison to the traditional method of Logistic Regression. In addition, it has be…
New method tests independence using ROC analysis and bipartite ranking.
problem Testing independence of two random variables with unknown marginals.
method Nonparametric framework based on ROC analysis and bipartite ranking.
result The method detects small departures from independence in high dimensions.
Geometric analysis of ROC and PR curves for binary classification.
problem Understanding classifier behavior and selection of optimal operating points.
method Geometric perspective on ROC and PR curves, focusing on the composition function G. result Many binary classification metrics are functions of G=Fp∘Fn−1, facilitating better classifier optimization. Deep learning uses ROC cost functions to improve virtual screening accuracy.
problem Challenges in training deep learning models for virtual screening, especially class imbalance and lack of ground truth labels.
method Proposes using ROC cost functions to optimize deep learning models for virtual screening, introduces new training schemes and cost functions.
result Demonstrates improved performance of ROC-based approaches on PubChem datasets.
Deep ROC analysis improves model selection and interpretation in medical and AI applications.
problem Inadequate performance measures for binary classifiers.
method Deep ROC analysis, translating AUC and partial AUC into balanced average accuracy and post-test measures.
result Deep ROC analysis provides balanced average accuracy, average sensitivity, and average specificity.
The variational autoencoder (VAE) is a generative model with continuous latent variables where a pair of probabilistic encoder (bottom-up) and decoder (top-down) is jointly learned by stochastic gradient variational Bayes. We first elaborate Gaussian VAE, approximating the local covariance matrix of the decoder as an o…
We investigate a long-debated question, which is how to create predictive models of recidivism that are sufficiently accurate, transparent, and interpretable to use for decision-making. This question is complicated as these models are used to support different decisions, from sentencing, to determining release on proba…
In many situations, the choice of an adequate similarity measure or metric on the feature space dramatically determines the performance of machine learning methods. Building automatically such measures is the specific purpose of metric/similarity learning. In Vogel et al. (2018), similarity learning is formulated as a …
New method detects adversarial attacks in industrial systems.
problem Adversarial attacks compromise machine learning models in industrial control systems.
method Introduces domain adaptation layer and Monte Carlo simulation for proactive defense.
result Approximates Bayes optimal mitigation for improved detection model health.
The performance of many machine learning techniques depends on the choice of an appropriate similarity or distance measure on the input space. Similarity learning (or metric learning) aims at building such a measure from training data so that observations with the same (resp. different) label are as close (resp. far) a…
New metrics fail adversarial tests, with some more robust than others.
problem Evaluation metrics for time-series anomaly detection were improved but not fully robust.
method Adversarial stress-testing of 12 adopted metrics on real benchmarks.
result Some metrics are more robust than others, with ROC-based metrics being gamed more often.
This paper visualizes uncertainty in classifier performance metrics.
problem Overemphasis on model performance metrics risks overlooking uncertainty.
method Developed visualizations of confusion matrix metric distributions.
result Uncertainty in performance metrics can overshadow model differences.
Dual-Channel Tensor Neural Network (DC-TNN) decomposes tensor data into low-rank and sparse components for better estimation and inference.
problem Tensor-valued data with multilinear dependencies are challenging to process due to loss of multiway geometry under vectorization.
method DC-TNN decomposes tensors into a low-rank core and a sparse refinement, processing them through coupled neural channels.
result Established non-asymptotic risk bounds and developed structure-aware conformal ROC and AUC confidence bands.
The area under the ROC curve (AUC) is a widely used performance measure in machine learning. Increasingly, however, in several applications, ranging from ranking to biometric screening to medicine, performance is measured not in terms of the full area under the ROC curve, but in terms of the \emph{partial} area under t…
Using the trends of estimated abilities in terms of item response theory for online testing, we can predict the success/failure status for the final examination to each student at early stages in courses. In prediction, we applied the newly developed nearest neighbor method for determining the similarity of learning sk…
New algorithm optimizes AUC in binary classification and changepoint detection.
problem Difficult to optimize AUC in binary classification and changepoint detection.
method Proposes efficient path-following algorithms for choosing optimal learning rate.
result Proposed line search algorithm computes complete AUM/AUC representation.
Paper proposes a new algorithm to minimize AUC disparities in machine learning models.
problem Minimizing unfairness in AUC scores for machine learning models.
method Proposes a minimax learning and bias mitigation framework for AUC optimization.
result Proves the convergence of the proposed algorithm to minimize group-level AUC.
Standardized fairness measures for continuous risk scores using Wasserstein distance.
problem Quantifying and interpreting group disparities in continuous risk scores.
method Proposes standardized fairness measures based on Wasserstein distance for continuous scores.
result Proposed measures outperform ROC-based fairness measures by being more explicit and quantifying significant biases.
The ROC curve is widely used to assess the quality of prediction/classification/ranking algorithms, and its properties have been extensively studied. The precision-recall (PR) curve has become the de facto replacement for the ROC curve in the presence of imbalance, namely where one class is far more likely than the oth…
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
AUC used as a measure of clustering quality in unsupervised learning.
problem Using AUC to evaluate clustering quality in unsupervised learning.
method Developed AUCC as a measure of clustering quality, derived theoretical expected values, and compared with Gamma criterion.
result AUCC has a consistent expected value under random clustering, making it a reliable measure.
Proposes and evaluates three diagnostic graphics for probabilistic classifiers.
problem Evaluating and comparing probabilistic classifiers.
method Triptych of diagnostic graphics: reliability diagram, ROC curve, Murphy diagram.
result Visual diagnostics reveal distinct aspects of forecast performance.
Assessing the performance of a learned model is a crucial part of machine learning. However, in some domains only positive and unlabeled examples are available, which prohibits the use of most standard evaluation metrics. We propose an approach to estimate any metric based on contingency tables, including ROC and PR cu…
Predicts accuracy of classifiers on unseen classes.
problem Unknown accuracy of classifiers on unseen classes.
method Defined rROC to estimate classifier accuracy on unseen classes.
result Robust algorithm CleaneX achieves better predictions.
Deep neural networks predict earthquake locations with high accuracy.
problem Predicting the location of earthquakes with high precision.
method Recurrent Convolutional Neural Networks (R-CNN) model that accounts for spatio-temporal dependencies.
result Neural networks model outperforms baseline models in predicting earthquakes with ROC AUC 0.975 and PR AUC 0.0890.
The paper proposes a test to assess rater accuracy while accounting for rater covariates.
problem Assessing the accuracy of raters in medical imaging and forensic studies.
method Covariate-adjusted homogeneity test to determine differences in accuracy among multiple rater groups.
result The proposed test identifies statistically significant differences among five participant groups in a face recognition study.