Paper assesses error estimates of Random Forests classification.
problem Quantitative assessment of Random Forests error estimates.
method Theoretical and empirical investigation of various error estimation methods.
result Random Forests' error estimates are closer to true error rate than average prediction error.
Estimates neural network errors for classification problems.
problem Binary and multi-class classification problems.
method Rademacher complexity estimates and direct approximation theorems.
result A priori error estimates for regularized loss functionals.
Efficient classifier error estimation without re-training.
problem Estimating classifier error without re-training.
method Generalized resubstitution based on empirical measures.
result Consistent and asymptotically unbiased error estimation.
Paper introduces a new method for error estimation in classification tasks with limited data.
problem Challenges in designing accurate classifiers and evaluating their performance with limited training data.
method Introduces a novel Bayesian MMSE estimator for optimal Bayesian transfer learning (OBTL) using Monte Carlo importance sampling.
result Proposed OBTL error estimation scheme outperforms standard methods, especially in small-sample settings.
Optimizes calibration error estimators for better classifier trustworthiness.
problem Lack of guidance on selecting and tuning calibration error estimators.
method Reformulates calibration estimation as a regression problem with i.i.d. input pairs.
result Demonstrates the effectiveness of optimized calibration estimators on image classification tasks.
This work bounds classification error in machine learning for low Bayes error conditions.
problem Understanding the error mismatch between Bayes error and model-based classification error.
method Applying classification error bounds to study the relationship with Kullback-Leibler divergence and proposing a linear approximation for low Bayes error conditions.
result A linear approximation of the classification error bound for low Bayes error conditions is proposed.
Meta learning of optimal classifier error rates allows an experimenter to empirically estimate the intrinsic ability of any estimator to discriminate between two populations, circumventing the difficult problem of estimating the optimal Bayes classifier. To this end we propose a weighted nearest neighbor (WNN) graph es…
A new method estimates Bayes error for deep networks, suggesting they may have reached the limit.
problem Evaluating the performance of deep learning models and detecting overfitting.
method A simple and direct Bayes error estimator based on uncertainty of class assignments.
result Deep networks may have reached the Bayes error limit for benchmark datasets.
Proposes improved classification via transfer learning with regularized linear discriminant analysis.
problem High dimensionality and small sample sizes lead to poor classification performance.
method Regularized random-effects linear discriminant analysis, combining ridge estimates from target and source models.
result Explicit derivation of asymptotic weights and classification error rates in high-dimensional settings.
Paper estimates optimal classification error with soft labels and calibration.
problem Estimating the optimal classification error with soft labels and calibration.
method Extends previous work on soft labels to estimate Bayes error, addressing bias and corrupted labels.
result The method provides a statistically consistent estimator of the Bayes error, even with imperfectly calibrated soft labels.
In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-…
Bottlenecks of binary classification from positive and unlabeled data (PU classification) are the requirements that given unlabeled patterns are drawn from the test marginal distribution, and the penalty of the false positive error is identical to the false negative error. However, such requirements are often not fulfi…
Multivariate pattern analyses approaches in neuroimaging are fundamentally concerned with investigating the quantity and type of information processed by various regions of the human brain; typically, estimates of classification accuracy are used to quantify information. While a extensive and powerful library of method…
We propose an empirical Bayes estimator based on Dirichlet process mixture model for estimating the sparse normalized mean difference, which could be directly applied to the high dimensional linear classification. In theory, we build a bridge to connect the estimation error of the mean difference and the misclassificat…
Bayes Error Rate estimators are evaluated for accuracy and sample requirements.
problem Evaluating the accuracy and sample requirements of Bayes Error Rate estimators.
method Monte Carlo simulations with synthetic data and real-world scenarios.
result k-Nearest Neighbor (kNN) is the most accurate non-parametric estimator.
Synthetic data augmentation can improve imbalanced classification metrics.
problem Improving imbalanced classification metrics
method Developing a framework for analyzing the effects of synthetic data augmentation on score-based classification
result Augmentation can improve AUROC, AUPRC, balanced accuracy, and F1 score
Estimates classifier errors without ground truth using algebraic geometry.
problem Lack of ground truth in real-world production systems.
method Non-parametric estimation using algebraic geometry to solve the self-assessment problem.
result Accuracy estimators are better than one part in a hundred.
Improves model calibration for deep neural networks using proper scores.
problem Calibration errors in deep neural networks are often biased and inconsistent.
method Introduces proper calibration errors related to proper scores.
result Demonstrates the superiority of proper scores over common estimators.
In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class…
Dual-T method improves transition matrix estimation in noisy label learning.
problem Large estimation error in noisy class posterior leads to poor transition matrix estimation.
method Introducing an intermediate class to avoid direct estimation of noisy class posterior, factorizing the transition matrix into two easier-to-estimate matrices.
result The dual-T estimator leads to better classification performances.
Optimizes classification algorithms with bounds on error rates.
problem Bounding uncertainties in classifier outputs for diagnostic testing.
method Set-theoretic and probabilistic arguments to derive uniform error bounds.
result Optimal partition minimizes the largest Gershgorin radius of the confusion matrix.
The paper analyzes classification algorithms on Korobov space and derives learning rates.
problem Analyzing classification performance on Korobov space.
method Tikhonov regularization and η-norm loss function for learning rates. result Derives learning rates for excess misclassification error in Korobov space.
We revisit resampling procedures for error estimation in binary classification in terms of U-statistics. In particular, we exploit the fact that the error rate estimator involving all learning-testing splits is a U-statistic. Thus, it has minimal variance among all unbiased estimators and is asymptotically normally dis…
New method corrects bias in density ratio estimation for missing data.
problem Missing data bias in density ratio estimation.
method Adapted KLIEP method (M-KLIEP) for MNAR data.
result M-KLIEP restores consistency and minimax optimality.
New method improves false-/true-positive-rate estimation in fraud detection with noisy labels.
problem Estimating FPR/TPR in fraud detection with class-conditional label noise.
method Directly cleaning model's validation data to de-correlate cleaning error with model scores.
result Improves accuracy of FPR/TPR estimates, especially in asymmetric label noise scenarios.
A statistical model predicts generalization in few-shot learning.
problem Lack of validation sets in few-shot learning makes generalization estimation difficult.
method Introduced a Gaussian model of feature distribution and an unbiased estimator for class-conditional density distances.
result Our approach outperforms alternatives like leave-one-out cross-validation.
A new method for high-dimensional data classification reduces misclassification errors.
problem High-dimensional data classification with limited samples.
method Compressive Regularized Discriminant Analysis (CRDA) using joint-sparsity promoting hard thresholding and regularized covariance matrix estimators.
result CRDA gives fewer misclassification errors than competitors and accurately selects features.
Develops high-dimensional measurement error models for non-linear loss functions.
problem Measurement errors in ultrahigh-dimensional biomedical data.
method Lipschitz loss functions, L1 norm minimization, Lasso analog.
result Improved accuracy in classification and quantile regression problems.
New weighted Lasso estimates improve logistic regression performance with measurement error.
problem Improper Lasso estimates in sparse logistic regression with equal penalties.
method Proposed weighted Lasso estimates using McDiarmid inequality for non-asymptotic oracle inequalities.
result Finite sample behavior illustrated by non-asymptotic oracle inequalities for estimation and prediction errors.
Binary classification improves with a small fraction of corrupted labels.
problem Binary classification with corrupted labels.
method Established corruption as a form of regularization and computed upper bounds on estimation error.
result Corruption is beneficial only up to a small fraction of the total sample, scaling with the square root of the sample size.
The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been…
Confidence measures for the generalization error are crucial when small training samples are used to construct classifiers. A common approach is to estimate the generalization error by resampling and then assume the resampled estimator follows a known distribution to form a confidence set [Kohavi 1995, Martin 1996,Yang…
This paper improves error estimation in covariate shift by incorporating target information.
problem Error estimation is inaccurate in covariate shift scenarios.
method Proposes a redefinition of importance using target information for better error estimation.
result Incorporating target information leads to more accurate error estimation, especially with KLIEP.
A method for safe online classification reduces test costs while maintaining low error rates.
problem Sequential testing for binary disease outcomes with unknown logistic model parameters.
method Joint estimation of logistic parameter and feature distribution with a conservative threshold.
result Achieves target error with high probability and requires minimal excess tests.
CRUDE calibrates regression uncertainty without assuming specific error distributions.
problem Uncalibrated uncertainty estimates in regression models, especially for modern predictive tasks.
method CRUDE assumes error distributions have a constant shape, shifted and scaled by predicted mean and standard deviation.
result CRUDE produces sharper, better calibrated, and more accurate uncertainty estimates than existing methods.
A novel method for classification with rejection using ensemble of cost-sensitive classifiers.
problem Avoid risky misclassification in error-critical applications.
method Learning an ensemble of cost-sensitive classifiers.
result Improved classification accuracy and flexibility in loss selection.
We propose a new splitting criterion for a meta-learning approach to multiclass classifier design that adaptively merges the classes into a tree-structured hierarchy of increasingly difficult binary classification problems. The classification tree is constructed from empirical estimates of the Henze-Penrose bounds on t…
ACA method improves gradient estimation for neural ODEs, reducing error and training time.
problem Inaccurate gradient estimation methods hinder the performance of neural ODEs on benchmark tasks.
method Adaptive Checkpoint Adjoint (ACA) method that applies trajectory checkpointing, deletes redundant components, and supports adaptive solvers.
result ACA reduces error rate by half and training time by half compared to adjoint and naive methods on image classification tasks.
Paper proposes methods to estimate minimal adversarial perturbations for deep neural networks.
problem Quantifying robustness of deep neural networks against adversarial attacks.
method Proposes two lightweight strategies to find minimal adversarial perturbation.
result Approximates theoretical distance for samples close to classification boundary, providing robustness guarantees.
New method estimates tensors from noisy data with missing entries.
problem Tensor estimation from noisy observations with missing entries.
method Sign series representation for tensor completion, addressing low- and high-rank signals.
result Excess risk bounds, estimation error rates, and sample complexities established.
New method recovers predictions from unobservable source subpopulation in binary classification.
problem Challenging binary classification with unobservable subpopulation in source domain.
method Distribution matching method to estimate subpopulation proportions, rigorous derivation of prediction models.
result Our method outperforms naive benchmarks in synthetic and real-world datasets.
Solves regression problems with CP by converting to classification.
problem Challenges in CP for heteroscedastic, multimodal, or skewed regression outputs.
method Converts regression to classification, uses CP for classification to obtain CP sets for regression.
result Simple approach yields good results on practical problems.
Develops NPMC method for noisy labels, improving multiclass classification accuracy.
problem Asymmetric misclassification costs and label noise in multiclass classification.
method Empirical likelihood approach using exponential tilting density ratio model.
result Root n consistent and asymptotically normal estimators for clean labels and noise mechanism.
Machine learning classification limits estimated using Kullback-Leibler divergence and Cohen's Kappa.
problem Estimating the best possible performance of machine learning classification algorithms.
method Relating Kullback-Leibler divergence to Cohen's Kappa and using the Chernoff-Stein Lemma to estimate error rates.
result Classification algorithms could not have performed any better due to underlying probability density functions for the two classes.
A method for making predictions with a reject option using conformal prediction.
problem Uncertainty in machine learning predictions, especially when models are unsure.
method Formalizing ML with reject option, using conformal prediction for distribution-free error guarantees.
result Theoretical guarantees on error rate for prediction sets with distribution-free validity.
Study non-asymptotic bounds for robust estimators under misspecified models.
problem Evaluate performance of robust estimators under adversarial conditions.
method Propose a general approach to adversarial risk analysis, including investigations on generalization and approximation errors.
result Establish non-asymptotic upper bounds for adversarial excess risk under Lipschitz loss functions.
This paper compares average-K and top-K classification methods under ambiguity.
problem Choosing a single label in ambiguous cases leads to low precision.
method Formally characterizes ambiguity profiles and compares average-K and top-K classification methods.
result Average-K can achieve lower error rates than top-K in some ambiguous cases.
The study reveals a linear relationship between source and target domain classification errors based on disagreement.
problem Evaluating model performance under distribution shift with limited labeled data.
method Developed a theoretical foundation for analyzing disagreement in high-dimensional random features regression.
result The disagreement-on-the-line phenomenon occurs when classification error under the source domain is a linear function of the target domain.