Study controls error rates of binary classifiers using hypothesis testing.
problem Traditional binary classifiers have uncontrolled error rates.
method Combines binary classification with statistical hypothesis testing.
result Trained classifiers can be made to meet target error rate thresholds.
Optimal classification rules control error rates in multiclass mixture models.
problem Classifying observations in multiclass mixture models while controlling error rates.
method Finding optimal classification rules by searching an optimal region in the observation space, using Maximum A Posteriori (MAP) rule and heuristic computation.
result The FDR-like optimal rule can be significantly less conservative than thresholded MAP rules.
This paper tackles worst-class error rate in classification tasks.
problem Minimizing worst-class error rate in classification tasks, especially in medical image classification.
method Designing a boosting approach to bound the worst-class error rate using Deep Neural Networks (DNNs).
result The proposed boosting approach lowers worst-class test error rates while avoiding overfitting.
Paper assesses error estimates of Random Forests classification.
problem Quantitative assessment of Random Forests error estimates.
method Theoretical and empirical investigation of various error estimation methods.
result Random Forests' error estimates are closer to true error rate than average prediction error.
Study shows exponential error reduction in multiclass classification without bias-variance trade-off.
problem Multiclass classification with margin conditions.
method Analysis of classification error under hard-margin conditions.
result Exponential decrease in classification error without bias-variance trade-off.
Characterizes distribution-free rates in unbalanced classification problems.
problem Minimizing error under two different distributions in unbalanced settings.
method Characterizes minimax rates over all pairs of distributions using a geometric condition.
result Identifies a dichotomy between hard and easy classes based on a three-points-separation condition.
The paper analyzes classification algorithms on Korobov space and derives learning rates.
problem Analyzing classification performance on Korobov space.
method Tikhonov regularization and η-norm loss function for learning rates. result Derives learning rates for excess misclassification error in Korobov space.
Study derives error decay rates for kernel classification under source and capacity conditions.
problem Understanding prediction error decay rates for real data sets.
method Derived decay rates for misclassification error under Gaussian design for SVM and ridge classification.
result Rates accurately describe learning curves for data sets satisfying source and capacity conditions.
Paper establishes a universal growth rate for smooth surrogate losses in classification.
problem Analyzing growth rates of consistency bounds for various surrogate losses.
method Proves square-root growth rate for smooth margin-based losses; extends to multi-class classification.
result Demonstrates a universal square-root growth rate for smooth comp-sum and constrained losses.
New bounds on majority voting's accuracy for multi-class classification problems.
problem Determining the accuracy of majority voting for multi-class classification.
method Analyzing the majority voting function under different voter conditions and distributions.
result The error rate of majority voting exponentially decays or grows with the number of voters under certain conditions.
This research examines how the error rate of nearest neighbor classifiers varies with dataset size.
problem The scaling of classification error rates with dataset size is not uniform.
method Theoretical analysis of nearest neighbor classifiers, focusing on early and late phases of dataset size.
result The error rate of nearest neighbor classifiers can have fine-grained rates depending on the dataset size and data distribution.
A method for making predictions with a reject option using conformal prediction.
problem Uncertainty in machine learning predictions, especially when models are unsure.
method Formalizing ML with reject option, using conformal prediction for distribution-free error guarantees.
result Theoretical guarantees on error rate for prediction sets with distribution-free validity.
We consider stochastic gradient descent and its averaging variant for binary classification problems in a reproducing kernel Hilbert space. In the traditional analysis using a consistency property of loss functions, it is known that the expected classification error converges more slowly than the expected risk even whe…
Meta learning of optimal classifier error rates allows an experimenter to empirically estimate the intrinsic ability of any estimator to discriminate between two populations, circumventing the difficult problem of estimating the optimal Bayes classifier. To this end we propose a weighted nearest neighbor (WNN) graph es…
Although kernel methods are widely used in many learning problems, they have poor scalability to large datasets. To address this problem, sketching and stochastic gradient methods are the most commonly used techniques to derive efficient large-scale learning algorithms. In this study, we consider solving a binary class…
OptiNet achieves near-minimax error rates with compression in Euclidean space.
problem Error and compression rates in non-parametric multiclass classification.
method Compression-based learning rule OptiNet and a novel general compression scheme.
result OptiNet achieves non-trivial compression rates with near-minimax error rates in Euclidean space.
We carefully study how well minimizing convex surrogate loss functions, corresponds to minimizing the misclassification error rate for the problem of binary classification with linear predictors. In particular, we show that amongst all convex surrogate losses, the hinge loss gives essentially the best possible bound, o…
This work bounds classification error in machine learning for low Bayes error conditions.
problem Understanding the error mismatch between Bayes error and model-based classification error.
method Applying classification error bounds to study the relationship with Kullback-Leibler divergence and proposing a linear approximation for low Bayes error conditions.
result A linear approximation of the classification error bound for low Bayes error conditions is proposed.
We show how to compute the Bayes error-rate for speaker verifiers.
problem How many errors does a speaker verifier make in a hundred trials?
method We compute the Bayes error-rate using calibrated likelihood ratios and user-supplied prior probabilities.
result The Bayes error-rate is upper bounded by the minimum of EER, P, and 1-P.
We consider binary classification problems with positive definite kernels and square loss, and study the convergence rates of stochastic gradient methods. We show that while the excess testing loss (squared loss) converges slowly to zero as the number of observations (and thus iterations) goes to infinity, the testing …
For binary classification we establish learning rates up to the order of n−1 for support vector machines (SVMs) with hinge loss and Gaussian RBF kernels. These rates are in terms of two assumptions on the considered distributions: Tsybakov's noise assumption to establish a small estimation error, and a new geometr…
In various approaches to learning, notably in domain adaptation, active learning, learning under covariate shift, semi-supervised learning, learning with concept drift, and the like, one often wants to compare a baseline classifier to one or more advanced (or at least different) strategies. In this chapter, we basicall…
Minimizes indecisions in selective classification to control misclassification rates.
problem Controlling misclassification rates in high-risk scenarios.
method Using indecisions to control misclassification rates, even below Bayes optimal.
result Control of misclassification rates to any user-specified level, even below Bayes optimal.
Ensembling improves performance when classifiers disagree more than average.
problem When do ensembles provide significant performance improvements in classification tasks?
method Theoretical and empirical analysis of ensemble improvement rate and disagreement-error ratio.
result Ensembling improves performance significantly when the disagreement rate is large relative to the average error rate.
New algorithm for adaptive experimental design in scientific settings.
problem Identifying true positives while controlling false discoveries in adaptive experimental design.
method Provably sample efficient adaptive algorithm for FDR control.
result First provably sample efficient adaptive algorithm for adaptive experimental design.
Paper improves classification rates for private data.
problem Classifying data with privacy constraints and relaxed assumptions.
method Introduced a novel approach for classification under privacy constraints, relaxing the strong density assumption.
result Achieved minimax optimal convergence rates without strong density assumption.
The study analyzes how machine learning classifiers' error rates decrease exponentially based on large deviations theory.
problem Understanding the convergence rate of machine learning classifiers' error probabilities.
method Large deviations theory applied to machine learning classification techniques.
result The error probability of ML classifiers converges to zero exponentially, with a rate dependent on the training set size.
This article studies the achievable guarantees on the error rates of certain learning algorithms, with particular focus on refining logarithmic factors. Many of the results are based on a general technique for obtaining bounds on the error rates of sample-consistent classifiers with monotonic error regions, in the real…
We propose a new splitting criterion for a meta-learning approach to multiclass classifier design that adaptively merges the classes into a tree-structured hierarchy of increasingly difficult binary classification problems. The classification tree is constructed from empirical estimates of the Henze-Penrose bounds on t…
We provide a new online learning algorithm which utilizes online passive-aggressive learning (PA) and total-error-rate minimization (TER) for binary classification. The PA learning establishes not only large margin training but also the capacity to handle non-separable data. The TER learning on the other hand minimizes…
New method improves false-/true-positive-rate estimation in fraud detection with noisy labels.
problem Estimating FPR/TPR in fraud detection with class-conditional label noise.
method Directly cleaning model's validation data to de-correlate cleaning error with model scores.
result Improves accuracy of FPR/TPR estimates, especially in asymmetric label noise scenarios.
Study on error probabilities of machine learning classification techniques using large deviations theory.
problem Performance analysis of machine learning binary classification techniques.
method Large deviations theory applied to Data-Driven Decision Function (D3F) for error probability analysis.
result Classification error probabilities vanish exponentially, with an asymptotic formula providing precise error rate estimates.
Decision makers increasingly rely on algorithmic risk scores to determine access to binary treatments including bail, loans, and medical interventions. In these settings, we reconcile two fairness criteria that were previously shown to be in conflict: calibration and error rate equality. In particular, we derive necess…
We empirically characterize the performance of discriminative and generative LSTM models for text classification. We find that although RNN-based generative models are more powerful than their bag-of-words ancestors (e.g., they account for conditional dependencies across words in a document), they have higher asymptoti…
The paper tackles fast rates in structured prediction problems.
problem Structured prediction problems with discrete outputs.
method Introducing continuous surrogate problems and leveraging their convergence rates for discrete problems.
result Super fast rates, including exponential rates, for excess risk in structured prediction problems.
Bayes Error Rate estimators are evaluated for accuracy and sample requirements.
problem Evaluating the accuracy and sample requirements of Bayes Error Rate estimators.
method Monte Carlo simulations with synthetic data and real-world scenarios.
result k-Nearest Neighbor (kNN) is the most accurate non-parametric estimator.
Theory extends optimal learning rates without realizability assumption.
problem Agnostic binary classification without realizability assumption.
method Identifies tetrachotomy of optimal rates and combinatorial structures.
result Optimal universal rates for binary classification in agnostic setting.
AdaBoost improves binary classification in robust one-bit compressed sensing with adversarial errors.
problem Binary classification in robust one-bit compressed sensing with adversarial errors.
method AdaBoost and max-ℓ1-margin-classifier approach, with convergence rates improved under certain feature conditions. result Improved convergence rates and explanation for harmless interpolating adversarial noise.
The paper develops bounds for predictive values in binary classification.
problem Lack of confidence intervals for positive and negative predictive values.
method Bi-criterion framework and distribution-free large deviation and uniform convergence bounds.
result New bounds for predictive values without relying on concentration inequalities.
In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-…
New PAC-Bayes bound controls multiple error types simultaneously.
problem Current PAC-Bayes bounds are limited to scalar metrics.
method Bounding KL divergence between empirical and true probabilities of multiple error types.
result First PAC-Bayes bound for rich information-rich certificates.
Develops a method to find costly high-confidence errors in black box models.
problem Finding rare high-confidence errors missed by random sampling.
method Adversarial perturbation-guided search technique to find errors at rates greater than expected given model confidence.
result Our Adversarial Distance search discovers high-confidence errors at a rate greater than expected given model confidence.
End-to-end ASR error detection using audio-transcript entailment.
problem Detecting transcription errors in ASR systems to prevent error propagation.
method Proposes a novel end-to-end approach using audio-transcript entailment, with acoustic and linguistic encoders.
result Achieves CER of 26.2% on all transcription errors and 23% on medical errors specifically, improving by 12% and 15.4% respectively over a strong baseline.
A method for safe online classification reduces test costs while maintaining low error rates.
problem Sequential testing for binary disease outcomes with unknown logistic model parameters.
method Joint estimation of logistic parameter and feature distribution with a conservative threshold.
result Achieves target error with high probability and requires minimal excess tests.
Improves risk control in predictions using semi-supervised calibration.
problem Noisy hyper-parameter tuning from limited labeled data.
method Semi-supervised calibration using unlabeled data to tune hyper-parameters rigorously.
result Improves prediction accuracy without sacrificing statistical validity.
This paper aims at refined error analysis for binary classification using support vector machine (SVM) with Gaussian kernel and convex loss. Our first result shows that for some loss functions such as the truncated quadratic loss and quadratic loss, SVM with Gaussian kernel can reach the almost optimal learning rate, p…
We propose a voted dual averaging method for online classification problems with explicit regularization. This method employs the update rule of the regularized dual averaging (RDA) method, but only on the subsequence of training examples where a classification error is made. We derive a bound on the number of mistakes…
Paper estimates optimal classification error with soft labels and calibration.
problem Estimating the optimal classification error with soft labels and calibration.
method Extends previous work on soft labels to estimate Bayes error, addressing bias and corrupted labels.
result The method provides a statistically consistent estimator of the Bayes error, even with imperfectly calibrated soft labels.