Two validation methods compared for classifier accuracy.
problem Validating trained classifiers using all in-sample data.
method SVOOSH vs. WAG (withhold and gap)
result WAG can be a better alternative for complex hypothesis classes and limited training data.
Ensemble validation shows selectivity penalties but variety benefits.
problem Selecting classifiers for ensemble models and their error bounds.
method Forming an ensemble from a set of hypothesis classifiers, selecting randomly, with an error bound formula.
result No penalty for using a richer hypothesis set if same fraction selected.
Bayesian model compares classifier accuracies across multiple datasets.
problem Shortcomings of null hypothesis significance tests in comparing classifier accuracies.
method Bayesian hierarchical model analyzing cross-validation results.
result Posterior probability of classifier accuracies being equivalent or different.
Study examines how different time series cross-validation methods affect anomaly detection in multivariate time series.
problem Evaluating anomaly detection in multivariate time series requires preserving temporal dependencies, especially for subsequence anomalies.
method Systematically investigates walk-forward and sliding window methods across various validation configurations and classifier types.
result Sliding window method consistently yields higher precision-recall scores and reduced fold-to-fold performance variance, particularly for deep learning models.
Prototype selection improves DS techniques' accuracy and reduces computational cost.
problem Improving the performance of dynamic selection techniques.
method Prototype selection techniques that edit validation data to remove noise and redundant instances.
result Improves DS techniques' classification accuracy and reduces computational cost.
The paper validates a classifier for identifying intraday regime shifts in MNQ futures.
problem Developing reliable trading signals from intraday regime shifts in MNQ futures.
method Constructed a composite day-classification system using three observable conditions.
result Classifier-positive days exhibit distinct intraday behavior but fail to generate profitable trading signals.
A new method combines classifiers using possibility distributions and adaptive t-norms.
problem Aggregating predictions from multiple classifiers trained on overlapping datasets.
method Proposes a new approach to aggregate classifier predictions using possibility theory and adaptive t-norms.
result Proves the proposed approach possesses desirable robustness properties.
The paper extends calibration to sets of probabilistic classifiers, finding many ensembles are poorly calibrated.
problem Evaluating the validity of epistemic uncertainty in sets of probabilistic classifiers.
method Proposed a novel nonparametric calibration test for sets of probabilistic classifiers.
result Ensembles of deep neural networks are often not well calibrated.
Develops methods for valid and validated confidence sets in multiclass and multilabel prediction.
problem Challenges of typical conformal prediction methods in multiclass and multilabel problems, especially uneven coverage.
method Leverages quantile regression to build methods that always guarantee correct coverage and asymptotically optimal conditional coverage, addressing label interactions with tree-structured classifiers.
result Empirical evaluation suggests more robust coverage of confidence sets.
Flat cross-validation is as good as nested CV for most classifiers.
problem Selecting the best classification algorithm and its hyperparameters.
method Used flat cross-validation (selecting hyperparams and estimating accuracy in one step) vs nested cross-validation (separate steps).
result Flat cross-validation is computationally cheaper and often selects similar quality algorithms as nested CV.
E-C2ST uses E-values for high-dimensional data two-sample tests.
problem Statistical testing for high-dimensional data.
method Combines split likelihood ratio tests and predictive independence tests, using E-values for anytime-valid sequential tests.
result E-C2ST achieves enhanced statistical power by partitioning datasets into multiple batches.
Paper investigates personalization in emotion recognition from physiological data.
problem Emotion recognition from physiological signals.
method Features are extracted and ranked based on their effect on classification accuracy. Different classifiers are compared. Inter-subject variability and personalization effect are investigated through trial-based and subject-based cross-validation. A personalized model is introduced.
result Personalized model enhances emotional state prediction.
Bootstrap aggregation, known as bagging, is one of the most popular ensemble methods used in machine learning (ML). An ensemble method is a ML method that combines multiple hypotheses to form a single hypothesis used for prediction. A bagging algorithm combines multiple classifiers modeled on different sub-samples of t…
Improved error bounds for classifiers using worst likely assignments.
problem Improving error bounds for accurate classifiers.
method Introduces a new statistic for permutation tests of worst likely assignments.
result Improves error bounds, especially for accurate classifiers.
Tests for classifier independence without ground truth labels.
problem Validation of classifier independence without ground truth labels.
method Exact solution for independent binary classifiers using algebraic geometry.
result Self-consistent test for classifier independence without ground truth labels.
SBSS uses similarity to split data for better classifier training.
problem Training better classifiers with realistic performance estimation.
method SBSS uses both input and output space information to split data using similarity functions.
result SBSS outperformed ordinary stratified 10-fold cross-validation in 75% of scenarios.
Machine learning detects survey validity from user behavior.
problem Detecting valid responses in web surveys.
method Uses mouse activity and machine learning models (LSTM, HMM).
result Predicts survey validity without analyzing specific answers.
Validates neural networks inputs to protect against adversarial examples.
problem Ensuring neural networks robustness against adversarial attacks.
method Runtime local robustness verification based on normal distribution of robustness radii.
result Improves neural network accuracy and protects against adversarial examples.
New criterion assesses cluster separability for validation.
problem Validating cluster analysis results and determining the number of clusters.
method Distinguishability criterion, combined loss function-based framework.
result Validated cluster configurations and determined the number of clusters.
Risk bounds for Classification and Regression Trees (CART, Breiman et. al. 1984) classifiers are obtained under a margin condition in the binary supervised classification framework. These risk bounds are obtained conditionally on the construction of the maximal deep binary tree and permit to prove that the linear penal…
SCRIB assigns multiple labels to each example to control class-specific prediction risks.
problem Lack of a sound mechanism to decide when to refrain from predicting in DL classifiers.
method Set-classifier with Class-specific Risk Bounds (SCRIB) that assigns multiple labels to each example and controls class-specific prediction risks.
result SCRIB obtained class-specific risks 35%-88% closer to the target risks than baseline methods.
Posterior conformal prediction improves prediction interval validity for subgroups.
problem Marginal and conditional prediction interval validity for subgroups.
method Modeling conditional nonconformity score distribution as a mixture of cluster distributions.
result PCP produces tighter prediction intervals, especially for well-represented clusters.
Optimizes signal detection in particle physics by decorrelating classifiers.
problem Systematic errors in background models can mislead signal detection.
method Use optimal transport to decorrelate classifiers from protected variables, then apply semiparametric mixture model.
result Decorrelation and signal enrichment improve the stability, robustness, and power of signal detection tests.
Deep learning model classifies drug effects based on structure and cell responses.
problem Limited ability to classify chemicals based on their modes of action.
method Integrative deep learning architecture combining molecular structures and cell responses.
result Improved classification performance, reducing error by 4.6%.
Improves classifier accuracy in ambiguous data settings.
problem Training classifiers with partially labeled data.
method Incremental pruning of candidate labels using conformal prediction.
result Significantly improves test set accuracies of PLL classifiers.
Crowdsourcing infers ground truth from multiple annotators, verified for supervised learning.
problem Obtaining universally valid ground truth for supervised learning is challenging and costly.
method Gather multiple annotations from diverse individuals, verify and aggregate for training classifiers.
result Inferred ground truth improves classifier performance in sensitive tasks like mitosis detection.
Combines classifiers from different types to improve ensemble accuracy.
problem Improving ensemble accuracy by combining classifiers of different types.
method Builds heterogeneous ensembles by pooling classifiers from multiple homogeneous ensembles, using cross-validation or out-of-bag data for optimal composition.
result Optimal heterogeneous ensemble compositions can be determined using cross-validation or out-of-bag data.
Vote-boosting uses weighted training data to build accurate and robust ensembles.
problem Generating accurate and robust ensemble classifiers.
method Sequential ensemble learning with weighted training data and emphasis on instances with high disagreement.
result Vote-boosting is effective for generating accurate and robust ensembles, especially when noise levels are low.
Study shows resampling labels improves classifier performance in noisy data.
problem Balancing sample size vs label reliability in noisy data.
method Comparing different validation strategies and analyzing MNIST database with varying noise levels.
result Classifier performance declines with high incorrect labels, highlighting the importance of resampling.
Efficient skin lesion analysis combines deep CNN and handcrafted features.
problem Skin lesion analysis for melanoma detection.
method Combines deep learning and handcrafted features, introduces hybrid features.
result 0.841 score on validation dataset using SVM classifier.
A new method combines simple binary classifiers to build complex multiclass classifiers, achieving performance limits in a Gaussian setting.
problem Building a sophisticated multiclass classifier from simple binary decisions.
method Combining O(logK) simple binary classifiers to form a K-class classifier. result Explicit performance bounds across various decoding and dimensional regimes for a stylized Gaussian setting.
Survey of methods for classifier comparison using precision.
problem Lack of methods for classifier comparison using precision.
method Statistical methods for precision comparison, accounting for inter-precision correlation.
result Methods to test global null hypothesis of model comparison using precision.
A new metric estimates classifier accuracy using only training data.
problem Assessing classifier accuracy without cross-validation.
method Bayesian Area Under the ROC Curve (CBAUC) metric for linear classifiers.
result The CBAUC is faster and more accurate than conventional AUC estimators.
The paper proposes sampling strategies for classifier copies.
problem Generating unlabelled points to explore decision behavior.
method Two sampling strategies compared with two standard methods.
result Validation in six problems and comparison of performance and cost.
A new cross-validation method reduces redundancy and improves model performance.
problem Redundancy in traditional k-fold cross-validation leads to biased results.
method Irredundant k-fold cross-validation, where each instance is used exactly once for training and testing.
result Consistent performance estimates with reduced variance and lower computational cost.
Bayesian neural networks improve performance at finite temperature.
problem Improving generalization in neural network classifiers.
method Sampling from finite temperature distributions derived from the posterior.
result Optimal performance achieved at non-zero temperature values.
New method uses synthetic data to validate financial agent classification.
problem Validation of machine learning methods for financial agent classification.
method Agent-based model to generate synthetic data for validation.
result Unsupervised clustering may give incorrect results for financial agents.
A statistical model predicts generalization in few-shot learning.
problem Lack of validation sets in few-shot learning makes generalization estimation difficult.
method Introduced a Gaussian model of feature distribution and an unbiased estimator for class-conditional density distances.
result Our approach outperforms alternatives like leave-one-out cross-validation.
The paper uses KKT conditions to reveal new insights into SVM behavior.
problem Understanding SVM behavior and tuning.
method Using Karush-Kuhn-Tucker conditions to explore SVM connections with other classifiers.
result SVM can be seen as a cropped version of mean difference and maximal data piling direction classifiers.
Improved KELM for multiclass classification with wavelet kernel.
problem Low test accuracy in multiclass classification problems.
method Mexican Hat wavelet kernel ELM.
result Significantly improved performance compared to other classifiers.
Novel method estimates CV-based classifier performance variance.
problem Lack of rigorous variance estimation methods for CV-based classifiers.
method Influence Function (IF) approach to estimate variance of CV-based estimators.
result IF-based method shows small RMS error with some bias, but ad-hoc methods still perform better.
Optimal model improves AUC, recall, and F1 score for class-imbalanced business risk.
problem Improving prediction of class-imbalanced business risk.
method Resampling, regularization, and model ensembling techniques.
result Boosting on DT with SMOTE oversampling achieves AUC, recall, and F1 score of 0.8633, 0.9260, and 0.8907, respectively.
Paper introduces SPADE method to protect classifiers from OOD and adversarial samples.
problem Protecting classifiers from out-of-distribution and adversarial samples.
method SPADE method based on GEV model in latent space.
result Provable protection against OOD and adversarial samples.
Dynamic classifier chains improve multi-label classification efficiency.
problem Building efficient multi-label classification models.
method Dynamic ensemble of chain classifiers using Naive Bayes and nearest neighbor approaches, with heuristic for label order optimization.
result The proposed dynamic chain model based on Naive Bayes classifier and heuristic is efficient for multi-label classification.
Optimizes SGD for text classification using hyper-parameter tuning.
problem Improving performance of SGD in text classification.
method Fine-tuning hyper-parameters using Grid-Search approach.
result Optimized SGD classification leads to better accuracy and execution time.
OTF uses optimal transport to measure classifier fairness.
problem Measuring and reducing unfairness in classifier predictions.
method Introduces Optimal Transport to Fairness (OTF) to quantify and reduce unfairness.
result OTF improves the balance between classifier performance and fairness.
Subject Cross Validation improves Human Activity Recognition performance by up to 16%.
problem Overestimation of Human Activity Recognition performance using k-fold cross validation.
method Investigated Subject Cross Validation vs. k-fold cross validation for Human Activity Recognition.
result Subject Cross Validation increases performance by up to 16%.
A new method distinguishes learning from memorization in DNNs using shuffled labels.
problem Differentiating learning from memorization in deep neural networks.
method Permutation approach using shared structure validation sets.
result DNNs can still learn from noisy inputs, not just memorize.