This paper introduces a new method to train normalizing flows using precision-recall divergences.
problem Training generative models with mode dropping and low-quality samples.
method Introduces PR-divergences and proposes a novel generative model to minimize precision-recall trade-offs.
result Normalizing flows can be trained to achieve specific precision-recall trade-offs using PR-divergences.
The study examines how class imbalance affects precision-recall curves.
problem Understanding how precision changes with class imbalance ratios.
method Analyzes the relationship between precision, class imbalance ratio, and true/false positive rates.
result Predicts changes in precision-recall curves and other measures with class imbalance ratios.
Revises precision-recall curves for generative models.
problem Improves evaluation of generative models by distinguishing mode-collapse and quality issues.
method Generalizes PR curve formulation to arbitrary measures, exposes a bridge to error rates, proposes a new algorithm to approximate precision-recall curves.
result Demonstrates the interest of the new formulation over the original approach on multi-modal datasets.
Proposes a new tree-based algorithm for class-imbalanced data.
problem Classification of imbalanced data.
method Precision-Recall Curve (PRC) classification tree algorithm.
result The PRC classification tree outperforms classic algorithms for class-imbalanced data.
Unified and extended precision-recall metrics for generative models.
problem Evaluation of generative models using scalar metrics like FID and IS.
method Unified various precision-recall approaches under a common framework.
result Recovered entire precision-recall curves and exposed metric pitfalls.
Enhanced anomaly detection using PRC-RF with autoencoders.
problem Extreme class imbalance and curse of dimensionality in anomaly detection.
method Hybrid framework combining PRC-RF and autoencoders.
result Autoencoder-PRC-RF model outperforms previous methods in accuracy, scalability, and interpretability.
Paper uses GANs for efficient rare disease detection.
problem Efficient detection of rare diseases with limited labeled data.
method Semi-supervised learning with GANs.
result Best precision-recall scores compared to baseline techniques.
Proposes a method to evaluate classifiers with missing labels using multiple imputation.
problem Missing labels during model evaluation can introduce bias, especially in Missing Not At Random (MNAR) data.
method Develops a multiple imputation technique to estimate and provide predictive distributions for metrics like precision, recall, and ROC-AUC.
result The predictive distribution's location and shape are generally correct, even in the MNAR regime.
Evaluation often aims to reduce the correctness or error characteristics of a system down to a single number, but that always involves trade-offs. Another way of dealing with this is to quote two numbers, such as Recall and Precision, or Sensitivity and Specificity. But it can also be useful to see more than this, and …
The paper proposes a method to calibrate metrics for better interpretability and fairness in machine learning models.
problem Machine learning metrics are biased by class priors and vary across subpopulations.
method Calibration of precision-based metrics (F1-score, AUC-PR) to make them invariant to class priors.
result Calibrated metrics improve interpretability and provide better control over model performance.
The paper critiques and expands on common evaluation metrics in machine learning.
problem The common evaluation metrics like Precision, Recall, F-Measure, and Rand Accuracy are biased and misleading.
method The paper introduces new measures like Informedness, Markedness, and Correlation to better reflect the quality of predictions.
result A system that performs worse in terms of Informedness can appear better using common measures like Precision and Recall.
Paper introduces a new evaluation framework for generative models using Rényi divergences.
problem Lack of tools to diagnose and assess generative models' performance.
method Develops a general evaluation framework using Rényi divergences to measure precision and recall.
result Extends existing techniques to continuous and discrete models with efficient algorithms.
New DTI model using self-attention molecule representation outperforms state-of-the-art.
problem Predicting drug-target interactions to reduce costs and improve personalized medicine.
method Proposes a new molecule representation using self-attention and a new DTI model.
result Our DTI model outperforms state-of-the-art by up to 4.9% points in precision-recall.
Reanalysis of bioactivity prediction models suggests SVM performance is competitive with deep learning.
problem Benchmarking and validation of machine learning models in drug discovery.
method Reanalysis of a large-scale comparison of machine learning models for bioactivity prediction, using numerical experiments to question ROC curve relevance and suggest precision-recall curve.
result Support vector machines show competitive performance with deep learning methods in bioactivity prediction.
Study evaluates three class imbalance techniques across diverse datasets.
problem Class imbalance in binary classification tasks.
method Synthetic Minority Over-sampling Technique (SMOTE), Class Weights tuning, Decision Threshold Calibration.
result Decision Threshold Calibration is the most consistently effective technique.
Study uses CNNs to detect sleep arousals more accurately.
problem Accurate detection of sleep arousals for better sleep quality assessment.
method Investigated five 1D CNN models on EEG signals from PhysioNet/Computing in Cardiology Challenge 2018.
result Best 1D CNN model achieved high precision and recall metrics.
Graphical models have gained a lot of attention recently as a tool for learning and representing dependencies among variables in multivariate data. Often, domain scientists are looking specifically for differences among the dependency networks of different conditions or populations (e.g. differences between regulatory …
New geometric metrics improve Bayesian optimization performance evaluation.
problem Current metrics lack geometric insights and cannot compare algorithms effectively.
method Proposed four geometric metrics: precision, recall, average degree, and average distance.
result Proposed metrics provide more detailed evaluation of Bayesian optimization.
Geometric analysis of ROC and PR curves for binary classification.
problem Understanding classifier behavior and selection of optimal operating points.
method Geometric perspective on ROC and PR curves, focusing on the composition function G. result Many binary classification metrics are functions of G=Fp∘Fn−1, facilitating better classifier optimization. In predictive maintenance, model performance is usually assessed by means of precision, recall, and F1-score. However, employing the model with best performance, e.g. highest F1-score, does not necessarily result in minimum maintenance cost, but can instead lead to additional expenses. Thus, we propose to perform model…
Framework for precise recall control in spatial conflation tasks.
problem Precise recall control in large-scale spatial conflation tasks to avoid downstream analytics failures and excessive manual review.
method End-to-end framework using equigrid bounding-box filter, CSR representation, neural ranker, and inverse-variance weighted ensemble of threshold estimators.
result Achieves exact recall with sub-percent variance over tens of millions of geometry pairs, runs on a single TPU v3 core.
AI random forest model improves credit risk scoring for Azerbaijani SMEs.
problem Improving accuracy in identifying defaulters for SME loans.
method Used Python to compare a Delphi model with a random forest model, measuring accuracy, precision, recall, and F-1 scores.
result Significant improvements in model performance (e.g., from 0.69 to 0.83 in accuracy).
Paper builds a supervised learning model for Chinese futures price prediction.
problem Predicting the trend of Chinese futures prices accurately.
method Supervised learning model designed for futures price movement classification.
result The model meets accuracy requirements for classifying futures price movements.
Producing overlapping schemes is a major issue in clustering. Recent proposed overlapping methods relies on the search of an optimal covering and are based on different metrics, such as Euclidean distance and I-Divergence, used to measure closeness between observations. In this paper, we propose the use of another meas…
Deep learning and lazy learner improve early sepsis detection.
problem Detecting sepsis early in high-resolution ICU records.
method Deep learning model with temporal convolutional network and Gaussian Process Adapter; lazy learner with dynamic time warping.
result Improves sepsis detection from 0.25 to 0.40 AUPRC 7 hours before onset.
Study develops algorithm to detect subtle respiratory events from PSG recordings.
problem Detecting non-apneic and non-hypopneic arousals in polysomnography recordings.
method Bidirectional LSTM classifier with 465 multi-domain features.
result 0.50 AUPRC on hidden test dataset, tied for second-best score in 2018 PhysioNet challenge.
Often the challenge associated with tasks like fraud and spam detection is the lack of all likely patterns needed to train suitable supervised learning models. This problem accentuates when the fraudulent patterns are not only scarce, they also change over time. Change in fraudulent pattern is because fraudsters contin…
A new active learning method for skewed data sets.
problem Severe class imbalance and small initial training data in sequential active learning.
method HAL: a hybrid active learning algorithm balancing labeled and unlabeled data.
result HAL makes better choices for labeling points than strong baselines.
Three RFF-based methods for nonlinear causal discovery in mixed data.
problem Nonlinear causal discovery in mixed data with computational constraints.
method FFML, TRFF, and FFCI methods for score-based, constraint-based, and hybrid causal discovery.
result FFML and TRFF methods provide complementary performance in causal discovery.
The optimal ranking score between precision and recall is rarely F1 and can be found using specific methods.
problem Finding a meaningful and optimal compromise between precision and recall scores.
method Established a shortest path between precision- and recall-induced rankings, framed the problem as an optimization problem, and provided theoretical tools to find the optimal β.
result F1 and its skew-insensitive version are not optimal tradeoffs between precision and recall scores.
TrackNet tracks tiny balls in sports videos with high precision.
problem Accurately tracking tiny and fast-moving objects like tennis balls in sports videos.
method Developed a heatmap-based deep learning network, TrackNet, trained on public and additional labeled videos.
result Achieved high precision (99.7%) in ball tracking on public domain videos.
Outlier detection methods have become increasingly relevant in recent years due to increased security concerns and because of its vast application to different fields. Recently, Pauwels and Lasserre (2016) noticed that the sublevel sets of the inverse Christoffel function accurately depict the shape of a cloud of data …
The presence of bacteria or fungi in the bloodstream of patients is abnormal and can lead to life-threatening conditions. A computational model based on a bidirectional long short-term memory artificial neural network, is explored to assist doctors in the intensive care unit to predict whether examination of blood cult…
Deep learning aids in automatic detection and segmentation of brain metastases.
problem Manual detection and segmentation of brain metastases is tedious and time-consuming.
method Deep learning approach using a fully convolutional neural network (CNN).
result Deep learning approach achieves high precision, recall, and Dice scores.
New metrics fail adversarial tests, with some more robust than others.
problem Evaluation metrics for time-series anomaly detection were improved but not fully robust.
method Adversarial stress-testing of 12 adopted metrics on real benchmarks.
result Some metrics are more robust than others, with ROC-based metrics being gamed more often.
When sufficient labeled data are available, classical criteria based on Receiver Operating Characteristic (ROC) or Precision-Recall (PR) curves can be used to compare the performance of un-supervised anomaly detection algorithms. However , in many situations, few or no data are labeled. This calls for alternative crite…
The study formalizes temporal precision and recall for anomaly detection in sequences.
problem Insufficient understanding of precision and recall in sequential anomaly detection.
method Formalized temporal precision and recall measures, developed time-tolerant confusion matrices, and demonstrated statistical significance.
result Precision and recall may overestimate performance with temporal tolerance.
Improved person detection in occluded conditions with AOS images.
problem Inaccurate classification of partially occluded people, animals, or objects.
method Combining multi-perspective images captured by Airborne Optical Sectioning (AOS) to achieve better precision and recall.
result Precision/recall of 96/93% for automated person detection.
Improved DDPMs achieve high log-likelihoods and sample quality with fewer passes.
problem Improving log-likelihoods of DDPMs while maintaining high sample quality.
method Modifying DDPMs with learning variances and using precision-recall metrics.
result DDPMs can achieve competitive log-likelihoods with fewer forward passes.
Machine learning predicts circulatory failure in ICU patients.
problem Limited ability of clinicians to recognize early signs of patient deterioration.
method Developed an early warning system using machine learning on ICU data.
result Predicts 90.0% of circulatory failure events with 81.8% identified more than two hours in advance.
In most machine learning applications, classification accuracy is not the primary metric of interest. Binary classifiers which face class imbalance are often evaluated by the Fβ score, area under the precision-recall curve, Precision at K, and more. The maximization of many of these metrics can be expressed as a con…
Study analyzes deep learning models for financial sentiment in earnings calls.
problem Leveraging NLP for sentiment analysis in financial transcripts.
method Comparative analysis of BERT, FinBERT, and ULMFiT models.
result Models' strengths and limitations in financial sentiment analysis.
A Support Vector Method for multivariate performance measures was recently introduced by Joachims (2005). The underlying optimization problem is currently solved using cutting plane methods such as SVM-Perf and BMRM. One can show that these algorithms converge to an eta accurate solution in O(1/Lambda*e) iterations, wh…
This paper proposes neural network-based undersampling techniques to improve model performance on class-imbalanced datasets.
problem Class imbalance problem in machine learning models leads to biased predictions and lower performance metrics.
method Neural network-based undersampling techniques applied to class-imbalanced datasets.
result Neural network-based undersampling outperforms other resampling techniques in terms of AUC, F1, and G-mean scores.
An attractive approach for fast search in image databases is binary hashing, where each high-dimensional, real-valued image is mapped onto a low-dimensional, binary vector and the search is done in this binary space. Finding the optimal hash function is difficult because it involves binary constraints, and most approac…
Proposes HBayes for hierarchical Bayesian recommendation learning.
problem Hierarchical structures in recommender systems.
method Hierarchical Bayesian learning framework with variational inference.
result Outperforms state-of-the-art models in precision, recall, and NDCG.
Entity resolution (ER) presents unique challenges for evaluation methodology. While crowdsourcing platforms acquire ground truth, sound approaches to sampling must drive labelling efforts. In ER, extreme class imbalance between matching and non-matching records can lead to enormous labelling requirements when seeking s…
Predicting highrisk vascular diseases is a significant issue in the medical domain. Most predicting methods predict the prognosis of patients from pathological and radiological measurements, which are expensive and require much time to be analyzed. Here we propose deep attention models that predict the onset of the hig…