A new metric estimates classifier accuracy using only training data.
problem Assessing classifier accuracy without cross-validation.
method Bayesian Area Under the ROC Curve (CBAUC) metric for linear classifiers.
result The CBAUC is faster and more accurate than conventional AUC estimators.
Reanalysis of bioactivity prediction models suggests SVM performance is competitive with deep learning.
problem Benchmarking and validation of machine learning models in drug discovery.
method Reanalysis of a large-scale comparison of machine learning models for bioactivity prediction, using numerical experiments to question ROC curve relevance and suggest precision-recall curve.
result Support vector machines show competitive performance with deep learning methods in bioactivity prediction.
Machine learning predicts circulatory failure in ICU patients.
problem Limited ability of clinicians to recognize early signs of patient deterioration.
method Developed an early warning system using machine learning on ICU data.
result Predicts 90.0% of circulatory failure events with 81.8% identified more than two hours in advance.
The paper proposes a test to assess rater accuracy while accounting for rater covariates.
problem Assessing the accuracy of raters in medical imaging and forensic studies.
method Covariate-adjusted homogeneity test to determine differences in accuracy among multiple rater groups.
result The proposed test identifies statistically significant differences among five participant groups in a face recognition study.
Many problems that appear in biomedical decision making, such as diagnosing disease and predicting response to treatment, can be expressed as binary classification problems. The costs of false positives and false negatives vary across application domains and receiver operating characteristic (ROC) curves provide a visu…
Study predicts antimicrobial resistance in ICU patients quickly.
problem Delayed AMR testing in ICU leads to suboptimal treatment.
method Developed predictive models using clinical and microbiological data.
result Machine learning models predict AMR with higher accuracy than naive model.
Predicts hospital patient discharge within 24 hours to optimize resource allocation.
problem Improving hospital resource management and patient care by prioritizing discharge.
method Used eight years of EHR data to train models predicting 24-hour discharge.
result Best models achieved AUROC of 0.85 and AUPRC of 0.53, well calibrated.
Paper compares WEMI target detection algorithms for weak signals.
problem Detecting weak magnetic signals in soil.
method Comparison of algorithms on two data sets.
result Strengths and weaknesses of compared approaches highlighted.
Receiver operating characteristic (ROC) analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under ROC curve (AUC) with standard cross-validation methods suffers from a large bias. The leave-pair-out (LPO) cross-validation has been shown to correct this bias. Howe…
Geometric analysis of ROC and PR curves for binary classification.
problem Understanding classifier behavior and selection of optimal operating points.
method Geometric perspective on ROC and PR curves, focusing on the composition function G. result Many binary classification metrics are functions of G=Fp∘Fn−1, facilitating better classifier optimization. New ROC tools assess predictive abilities for any linearly ordered outcomes.
problem Fundamental restriction in ROC analysis for non-dichotomous outcomes.
method ROC movies and UROC curves for linearly ordered outcomes.
result CPA equals AUC for binary outcomes and relates to Spearman's coefficient for pairwise distinct outcomes.
Proposes a new prior for VAEs to improve out-of-distribution detection.
problem Probabilistic generative models struggle with out-of-distribution detection.
method Introduces an exponentially tilted Gaussian prior for VAEs.
result Achieves state-of-the-art results on ROC-AUC metric.
A new method QMS22 for semi-supervised anomaly detection outperforms existing methods.
problem Semi-supervised anomaly detection in datasets with overlapping normal and outlier samples.
method QMS22, a classifier that solves a multi-class classification problem involving both training and test sets.
result QMS22 significantly outperforms ISOF and ocSVM in anomaly detection.
The paper analyzes how incentives and information affect ROC curves in medical diagnosis.
problem Understanding how incentives and information asymmetry influence ROC curves in medical diagnosis.
method Theoretical analysis and illustration with a real-world dataset.
result Different models of incentives and information asymmetry affect the relationship between human decisions and ROC curves.
LxCIM metric improves binary classification performance evaluation.
problem Evaluation metrics for binary classification are often not invariant to local class exchange.
method Proposes LxCIM, a rank-based metric invariant to local class exchange.
result LxCIM addresses limitations of existing metrics like AUROC.
Unified four trade-off curves for assessing generative model proximity.
problem Quantitative assessment of proximity between two probability distributions.
method Unified four existing curves: PR, Lorenz, ROC, and Rényi divergence frontiers.
result Explicit relationship between PR and Lorenz curves with domain adaptation bounds.
When sufficient labeled data are available, classical criteria based on Receiver Operating Characteristic (ROC) or Precision-Recall (PR) curves can be used to compare the performance of un-supervised anomaly detection algorithms. However , in many situations, few or no data are labeled. This calls for alternative crite…
Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
Deep learning has significant potential for medical imaging. However, since the incident rate of each disease varies widely, the frequency of classes in a medical image dataset is imbalanced, leading to poor accuracy for such infrequent classes. One possible solution is data augmentation of infrequent classes using syn…
We examine the supervised learning problem in its continuous setting and give a general optimality condition through techniques of functional analysis and the calculus of variations. This enables us to solve the optimality condition for the desired function u numerically and make several comparisons with other widely u…
Deep learning diagnoses MS from smartphone data.
problem Diagnosing MS with complex clinical assessments and tests.
method Deep-learning approach using smartphone-derived digital biomarkers.
result Deep-learning models distinguish MS with 88% accuracy.
High-throughput machine learning predicts thousands of diagnosis codes with high accuracy.
problem Predicting disease risk for thousands of diagnosis codes at various time points.
method Training machine learning algorithms on EHR data to predict diagnosis risks.
result Achieved AUCs of 0.803 and 0.758 for 1 and 6-month predictions, respectively.
In many healthcare settings, intuitive decision rules for risk stratification can help effective hospital resource allocation. This paper introduces a novel variant of decision tree algorithms that produces a chain of decisions, not a general tree. Our algorithm, α-Carving Decision Chain (ACDC), sequentially carves o…
Imbalanced data sets containing much more background than signal instances are very common in particle physics, and will also be characteristic for the upcoming analyses of LHC data. Following up the work presented at ACAT 2008, we use the multivariate technique presented there (a rule growing algorithm with the meta-m…
Hidden Markov Models classify cough events with high accuracy.
problem Identifying coughing events in noisy environments for health monitoring.
method Used Hidden Markov Models (HMMs) for cough classification.
result Multivariate HMMs achieved 92% AUR in classifying cough events.
Optimal cutoff interval for risk scores improves binary classification accuracy.
problem Improving binary classification accuracy with abstention.
method Determines optimal cutoff interval for risk scores, refraining from decisions outside this interval.
result Minimizes classification margin and maximizes accuracy within the interval.
Machine learning improves early detection of patient deterioration in Brazilian hospitals.
problem Challenges in recognizing clinical deterioration in hospital settings.
method Application of machine learning to analyze EHR data from multiple hospitals.
result Machine learning models outperformed traditional protocols by 25 percentage points in AUC.
Paper discusses extending Gini score for tied rankings and case weights.
problem Extending Gini score for tied rankings and case weights.
method Discuss and adapt Gini score for ties and case weights.
result Gini score can be used for tied rankings and case weights.
Models predict drug interactions with high accuracy.
problem Detecting drug-drug interactions to prevent medical injuries.
method Artificial neural networks and graph similarity measures.
result Models achieve high accuracy in predicting drug interactions.
Paper extends RPD for better handling multiple modalities and non-convexity.
problem Handling multiple modalities and non-convexity in data clouds.
method Computes RPD in a reproducing kernel Hilbert space using kernel principal component analysis.
result The method outperforms RPD and is comparable to other models on benchmark datasets.
Area under the receiver operating characteristics curve (AUC) is an important metric for a wide range of signal processing and machine learning problems, and scalable methods for optimizing AUC have recently been proposed. However, handling very large datasets remains an open challenge for this problem. This paper prop…
Neural network models improve ROC curve evaluation of biomarkers, focusing on age's role in physical activity-mortality association.
problem Improving biomarker evaluation using machine learning for complex relationships.
method Proposes neural network-based covariate-adjusted ROC modeling.
result Age has distinct effects on mortality outcomes when physical activity is measured as total activity time.
Study evaluates data augmentation methods for prostate cancer detection in MRI.
problem Limited data for prostate cancer detection in MRI.
method Static application of five augmentation techniques (rotation, flip, crop, translation) to MRI dataset.
result Rotation method improved 2D slice-based AUC to 0.85.
This paper visualizes uncertainty in classifier performance metrics.
problem Overemphasis on model performance metrics risks overlooking uncertainty.
method Developed visualizations of confusion matrix metric distributions.
result Uncertainty in performance metrics can overshadow model differences.
The paper explores symmetric losses for better learning from corrupted labels.
problem Learning from corrupted labels with balanced error rate or AUC maximization.
method Proves theoretical properties of symmetric losses and proposes a convex barrier hinge loss.
result Symmetric losses are advantageous in BER minimization and AUC maximization from corrupted labels.
A common approach in positive-unlabeled learning is to train a classification model between labeled and unlabeled data. This strategy is in fact known to give an optimal classifier under mild conditions; however, it results in biased empirical estimates of the classifier performance. In this work, we show that the typi…
Federated learning clusters EMRs to improve mortality and stay predictions.
problem Decentralized, non-IID EMRs complicate centralized machine learning algorithms.
method Introduced CBFL algorithm that clusters EMRs into communities for federated learning.
result CBFL outperforms baseline FL algorithm in ROC AUC, PR AUC, and communication cost.
Early detection of breast cancer can increase treatment efficiency. Architectural Distortion (AD) is a very subtle contraction of the breast tissue and may represent the earliest sign of cancer. Since it is very likely to be unnoticed by radiologists, several approaches have been proposed over the years but none using …
We present the qGaussian generalization of the Merton framework, which takes into account slow fluctuations of the volatility of the firms market value of financial assets. The minimal version of the model depends on the Tsallis entropic parameter q and the generalized distance to default. The empirical foundation and …
Proposes and evaluates three diagnostic graphics for probabilistic classifiers.
problem Evaluating and comparing probabilistic classifiers.
method Triptych of diagnostic graphics: reliability diagram, ROC curve, Murphy diagram.
result Visual diagnostics reveal distinct aspects of forecast performance.
New algorithm optimizes AUC in binary classification and changepoint detection.
problem Difficult to optimize AUC in binary classification and changepoint detection.
method Proposes efficient path-following algorithms for choosing optimal learning rate.
result Proposed line search algorithm computes complete AUM/AUC representation.
We investigate a long-debated question, which is how to create predictive models of recidivism that are sufficiently accurate, transparent, and interpretable to use for decision-making. This question is complicated as these models are used to support different decisions, from sentencing, to determining release on proba…
The paper develops a test for EU portfolio efficiency in high dimensions.
problem Testing the efficiency of the EU portfolio in high-dimensional settings.
method Shrinkage-based approach for portfolio weights and random matrix theory.
result Asymptotic behavior of the test statistic under high-dimensional conditions.
The consultative papers for the Basel II Accord require rating systems to provide a ranking of obligors in the sense that the rating categories indicate the creditworthiness in terms of default probabilities. As a consequence, the default probabilities ought to present a monotonous function of the ordered rating catego…
New optimization method improves AUC for binary classification and changepoint detection.
problem Non-convex AUC and sub-optimal points in ROC curves.
method AUM (Area Under Min(FP, FN)) surrogate loss function based on sorting and summing ROC curve points.
result AUM minimization learning algorithm improves AUC and speeds up compared to previous methods.
Study predicts startup outcomes like funding, patenting, IPOs using machine learning.
problem Forecasting startup success metrics like funding, patenting, IPOs.
method Developed interpretable machine learning framework, used preprocessing, class imbalance handling, and compared multiple models.
result Achieved high AUROC values for patent, funding, and exit predictions.
Knoop enhances variable selection with over-parameterization and knockoffs.
problem Challenges of variable selection in high-dimensional datasets.
method Generates knockoff variables, integrates them into an over-parameterized model, and uses anomaly-based significance tests.
result Superior performance in variable selection compared to existing methods.
A framework for document classification using keywords and unlabeled data.
problem Classifying documents without labels but with relevant keywords.
method Theoretical framework for optimizing AUC and other metrics.
result Effective use of keywords and unlabeled data for classification.