Paper estimates optimal ROC curve arc length and AUC, improving classification performance.
problem Estimating optimal ROC curve arc length and AUC in imbalanced binary classification.
method Expresses arc length and AUC as variational objectives, estimating using positive and negative samples.
result Proposed classification procedure maximizes an approximate lower bound of maximal AUC.
New optimization method improves AUC for binary classification and changepoint detection.
problem Non-convex AUC and sub-optimal points in ROC curves.
method AUM (Area Under Min(FP, FN)) surrogate loss function based on sorting and summing ROC curve points.
result AUM minimization learning algorithm improves AUC and speeds up compared to previous methods.
A scalable ROC-SVM variant reduces training time for imbalanced binary classification.
problem High computational cost of ROC-SVM for imbalanced binary classification.
method Incomplete U-statistics and low-rank kernel approximation.
result Comparable AUC performance with reduced training time.
CP-ROC bands improve graph classification accuracy and uncertainty quantification.
problem Uncertainty quantification and robustness to distributional shifts in graph classification.
method Conditional Prediction ROC (CP-ROC) bands for graph classification, developed for TGNNs and adaptable to GNNs.
result Statistically guaranteed coverage for CP-ROC under local exchangeability condition, improving prediction reliability.
Geometric analysis of ROC and PR curves for binary classification.
problem Understanding classifier behavior and selection of optimal operating points.
method Geometric perspective on ROC and PR curves, focusing on the composition function G. result Many binary classification metrics are functions of G=Fp∘Fn−1, facilitating better classifier optimization. Rapid overlay of chemical structures (ROCS) is a standard tool for the calculation of 3D shape and chemical ("color") similarity. ROCS uses unweighted sums to combine many aspects of similarity, yielding parameter-free models for virtual screening. In this report, we decompose the ROCS color force field into "color com…
New ROC tools assess predictive abilities for any linearly ordered outcomes.
problem Fundamental restriction in ROC analysis for non-dichotomous outcomes.
method ROC movies and UROC curves for linearly ordered outcomes.
result CPA equals AUC for binary outcomes and relates to Spearman's coefficient for pairwise distinct outcomes.
The area under the ROC curve (AUC) is a widely used performance measure in machine learning. Increasingly, however, in several applications, ranging from ranking to biometric screening to medicine, performance is measured not in terms of the full area under the ROC curve, but in terms of the \emph{partial} area under t…
New method tests independence using ROC analysis and bipartite ranking.
problem Testing independence of two random variables with unknown marginals.
method Nonparametric framework based on ROC analysis and bipartite ranking.
result The method detects small departures from independence in high dimensions.
Novel method for multiclass ROC curves using multidimensional Gini index.
problem Multiclass performance evaluation, especially for imbalanced datasets.
method Extends ROC curve methodology to multiclass settings using multidimensional Gini index.
result Validated through case studies in health care and finance.
New algorithm optimizes AUC in binary classification and changepoint detection.
problem Difficult to optimize AUC in binary classification and changepoint detection.
method Proposes efficient path-following algorithms for choosing optimal learning rate.
result Proposed line search algorithm computes complete AUM/AUC representation.
Paper proposes a new algorithm to minimize AUC disparities in machine learning models.
problem Minimizing unfairness in AUC scores for machine learning models.
method Proposes a minimax learning and bias mitigation framework for AUC optimization.
result Proves the convergence of the proposed algorithm to minimize group-level AUC.
Deep learning uses ROC cost functions to improve virtual screening accuracy.
problem Challenges in training deep learning models for virtual screening, especially class imbalance and lack of ground truth labels.
method Proposes using ROC cost functions to optimize deep learning models for virtual screening, introduces new training schemes and cost functions.
result Demonstrates improved performance of ROC-based approaches on PubChem datasets.
Deep ROC analysis improves model selection and interpretation in medical and AI applications.
problem Inadequate performance measures for binary classifiers.
method Deep ROC analysis, translating AUC and partial AUC into balanced average accuracy and post-test measures.
result Deep ROC analysis provides balanced average accuracy, average sensitivity, and average specificity.
Receiver operating characteristic (ROC) analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under ROC curve (AUC) with standard cross-validation methods suffers from a large bias. The leave-pair-out (LPO) cross-validation has been shown to correct this bias. Howe…
The fuzzy ROC extends Receiver Operating Curve (ROC) visualization to the situation where some data points, falling in an indeterminacy region, are not classified. It addresses two challenges: definition of sensitivity and specificity bounds under indeterminacy; and visual summarization of the large number of possibili…
The study improves the assessment of fairness in face recognition using ROC curves and statistical guarantees.
problem Improving the assessment of fairness in face recognition systems.
method Proves asymptotic guarantees for empirical ROC curves and fairness metrics, and introduces a recentering technique to avoid bootstrap pitfalls.
result Demonstrates the practical relevance of the methods for assessing fairness in face recognition systems.
Develops a new method for statistical optimal allocation problems.
problem Statistical optimal allocation problems with constraints.
method Functional differentiability approach and Hadamard differentiability of value functions.
result Validates margin assumption for fast convergence rate of plug-in methods.
The Receiver Operating Characteristic (ROC) curve is a representation of the statistical information discovered in binary classification problems and is a key concept in machine learning and data science. This paper studies the statistical properties of ROC curves and its implication on model selection. We analyze the …
Area under ROC (AUC) is an important metric for binary classification and bipartite ranking problems. However, it is difficult to directly optimizing AUC as a learning objective, so most existing algorithms are based on optimizing a surrogate loss to AUC. One significant drawback of these surrogate losses is that they …
Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
Improves ROC/AUC for multi-class classification.
problem Lack of sensible plots, sensitivity to imbalanced data, inability to specify mis-classification cost, and lack of evaluation uncertainty quantification.
method Factorizes multi-class ROC into a one-dimensional vector representation for visualization and summary.
result Provides a binary AUC-equivalent summary and mis-classification weights specification.
Neural network models improve ROC curve evaluation of biomarkers, focusing on age's role in physical activity-mortality association.
problem Improving biomarker evaluation using machine learning for complex relationships.
method Proposes neural network-based covariate-adjusted ROC modeling.
result Age has distinct effects on mortality outcomes when physical activity is measured as total activity time.
Optimizes partial AUC across various FPRs for machine learning models.
problem Lack of scalable algorithms for optimizing partial AUC in a range of FPRs.
method Formulated as a non-smooth DC program, developed an efficient approximated gradient descent method using Moreau envelope smoothing.
result Achieved a complexity of O(1/ε6) for finding nearly ε-critical solutions. The paper proposes tree-based methods for automatically learning similarity measures.
problem Automatically learning similarity measures in feature spaces.
method Formulates similarity learning as a pairwise bipartite ranking problem and uses recursive tree-based ROC optimization.
result Validates iterative partitioning procedures for similarity learning and proposes efficient algorithms.
ABROCA assesses algorithmic bias, revealing skewed distributions that inflate results.
problem Detecting nuanced performance differences in classifier fairness.
method Study of ABROCA metric's statistical properties under various conditions.
result ABROCA distributions are skewed, inflating results by chance in imbalanced classes.
Many problems that appear in biomedical decision making, such as diagnosing disease and predicting response to treatment, can be expressed as binary classification problems. The costs of false positives and false negatives vary across application domains and receiver operating characteristic (ROC) curves provide a visu…
Proposes MCC-F1 curve for better binary classification evaluation.
problem Misleading performance evaluations with ROC and PR curves for imbalanced data.
method Introduces MCC-F1 curve combining MCC and F1 score.
result MCC-F1 curve provides clearer classifier differentiation.
The paper tackles fairness in scoring functions for binary classification.
problem Fairness in scoring functions for binary classification tasks.
method Introduces ROC-based fairness constraints and learning algorithms.
result Generalization bounds and practical learning algorithms for fair scoring functions.
Feature selection aims to select the smallest subset of features for a specified level of performance. The optimal achievable classification performance on a feature subset is summarized by its Receiver Operating Curve (ROC). When infinite data is available, the Neyman- Pearson (NP) design procedure provides the most e…
Optimal cutoff interval for risk scores improves binary classification accuracy.
problem Improving binary classification accuracy with abstention.
method Determines optimal cutoff interval for risk scores, refraining from decisions outside this interval.
result Minimizes classification margin and maximizes accuracy within the interval.
Proposes and evaluates three diagnostic graphics for probabilistic classifiers.
problem Evaluating and comparing probabilistic classifiers.
method Triptych of diagnostic graphics: reliability diagram, ROC curve, Murphy diagram.
result Visual diagnostics reveal distinct aspects of forecast performance.
The variational autoencoder (VAE) is a generative model with continuous latent variables where a pair of probabilistic encoder (bottom-up) and decoder (top-down) is jointly learned by stochastic gradient variational Bayes. We first elaborate Gaussian VAE, approximating the local covariance matrix of the decoder as an o…
The paper proposes using theoretical ROC curves to categorize classifier responses.
problem The lack of explicit probability distributions for classifier responses in machine learning.
method Fit beta distributions to classifier responses and use them to categorize responses into different classes.
result Established a categorization of classifier responses into classes with different ROC curve extremal behaviors.
The performance of many machine learning techniques depends on the choice of an appropriate similarity or distance measure on the input space. Similarity learning (or metric learning) aims at building such a measure from training data so that observations with the same (resp. different) label are as close (resp. far) a…
DCC separates marginal estimation from dependence modeling for improved classification accuracy.
problem Classifying with strong dependence between features.
method Deep Copula Classifier using neural copula densities.
result Achieves excess-risk O(n−r/(2r+d)) for r-smooth copulas. We investigate a long-debated question, which is how to create predictive models of recidivism that are sufficiently accurate, transparent, and interpretable to use for decision-making. This question is complicated as these models are used to support different decisions, from sentencing, to determining release on proba…
New bounds for Neyman-Pearson region using f-divergences.
problem Bounding the Neyman-Pearson region for hypothesis testing.
method Establishing novel lower and upper bounds using f-divergences. result Best possible lower bound for the Neyman-Pearson boundary using hockey-stick f-divergences. New method improves fairness of facial recognition systems.
problem Facial recognition systems exhibit bias across different demographic groups.
method Optimizes centroid-based scores to reduce bias in pre-trained models.
result Demonstrates significant improvement in fairness with minimal loss in accuracy.
New classifiers ensure fairness by adjusting a base classifier's operating characteristics.
problem Ensuring fairness in binary classification with multiple group constraints.
method Intervening directly on a base classifier's operating characteristics using group-wise ROC convex hulls and post-processing.
result Methods satisfy multiple fairness constraints (DP, EO, PP) with minimal interventions and near-oracle accuracy.
Random forest model predicts sewer pipe deterioration with high accuracy.
problem Challenges in predicting and scheduling sewer pipe inspections.
method Random forest classification model for sewer pipe condition prediction.
result Model achieved excellent AUC of 0.81 in a case study for City of LA.
The study examines how verifier imperfections impact test-time scaling techniques.
problem Understanding how verifier imperfections affect test-time scaling methods.
method Proves the instance-level accuracy of Best-of-N and Rejection Sampling methods using the geometry of the verifier's ROC curve.
result RS outperforms BoN for fixed compute, but both converge to the same accuracy in the infinite-compute limit.
New metrics fail adversarial tests, with some more robust than others.
problem Evaluation metrics for time-series anomaly detection were improved but not fully robust.
method Adversarial stress-testing of 12 adopted metrics on real benchmarks.
result Some metrics are more robust than others, with ROC-based metrics being gamed more often.
The receiver operating characteristic (ROC) curve is a very useful tool for analyzing the diagnostic/classification power of instruments/classification schemes as long as a binary-scale gold standard is available. When the gold standard is continuous and there is no confirmative threshold, ROC curve becomes less useful…
This paper visualizes uncertainty in classifier performance metrics.
problem Overemphasis on model performance metrics risks overlooking uncertainty.
method Developed visualizations of confusion matrix metric distributions.
result Uncertainty in performance metrics can overshadow model differences.
Adequate evaluation of an information retrieval system to estimate future performance is a crucial task. Area under the ROC curve (AUC) is widely used to evaluate the generalization of a retrieval system. However, the objective function optimized in many retrieval systems is the error rate and not the AUC value. This p…
New method detects adversarial attacks in industrial systems.
problem Adversarial attacks compromise machine learning models in industrial control systems.
method Introduces domain adaptation layer and Monte Carlo simulation for proactive defense.
result Approximates Bayes optimal mitigation for improved detection model health.
TrialGraph uses graph machine learning to improve clinical trial design and predict side effects.
problem Complexity and cost in clinical trials hinder drug development.
method Curated clinical trial data set converted to graph-structured formats, applied graph machine learning algorithms.
result MetaPath2Vec algorithm performed exceptionally well, improving prediction accuracy.