This paper shows faster convergence rates for stochastic gradient descent in binary classification.
problem Achieving faster convergence rates for stochastic gradient descent in binary classification.
method Stochastic gradient descent and averaging variant, focusing on exponential convergence rates under strong low-noise conditions.
result Exponential convergence of the expected classification error in the final phase of stochastic gradient descent and averaged stochastic gradient descent for differentiable convex loss functions.
Sharp bounds on uniform generalization errors in binary linear classification.
problem Understanding the uniform generalization errors in binary linear classification.
method Isoperimetric arguments, Poincaré and log-Sobolev inequalities for joint distributions.
result Sharp concentration bounds on uniform generalization errors, almost sure convergence in broad settings.
Paper assesses error estimates of Random Forests classification.
problem Quantitative assessment of Random Forests error estimates.
method Theoretical and empirical investigation of various error estimation methods.
result Random Forests' error estimates are closer to true error rate than average prediction error.
Study shows exponential convergence in classification errors using random features and SGD.
problem Scalability issues in kernel methods for large datasets.
method Binary classification problem with random features and stochastic gradient descent.
result Exponential convergence rate of expected classification error achieved.
Develops a method to find costly high-confidence errors in black box models.
problem Finding rare high-confidence errors missed by random sampling.
method Adversarial perturbation-guided search technique to find errors at rates greater than expected given model confidence.
result Our Adversarial Distance search discovers high-confidence errors at a rate greater than expected given model confidence.
Bias - variance decomposition of the expected error defined for regression and classification problems is an important tool to study and compare different algorithms, to find the best areas for their application. Here the decomposition is introduced for the survival analysis problem. In our experiments, we study bias -…
New algorithm for Gaussian process classification using posterior linearisation.
problem Improving Gaussian process classification performance.
method Posterior linearisation to approximate posterior density iteratively, accounting for linearisation error.
result PL has better performance than EP in experimental data.
Study shows MDA's effectiveness even when more components are assumed than in actual data.
problem Classification error in overspecified Mixture Discriminant Analysis.
method Two-component Gaussian mixture model, EM algorithm, theoretical analysis of convergence and error rates.
result EM algorithm converges exponentially fast to Bayes risk with suitable initialization.
The paper analyzes the expected size of conformal prediction sets.
problem Lack of finite-sample analysis and guarantees for prediction set sizes.
method Theoretical quantification and empirical computation of expected set size.
result Derives point estimates and high-probability interval bounds for prediction set size.
State-level minimum Bayes risk (sMBR) training has become the de facto standard for sequence-level training of speech recognition acoustic models. It has an elegant formulation using the expectation semiring, and gives large improvements in word error rate (WER) over models trained solely using cross-entropy (CE) or co…
New calibration measures for multi-class classification improve model accuracy.
problem Calibration of multi-class classification models is insufficient for safety-critical applications.
method Developed new calibration measures and estimators for multi-class classification.
result Proposed estimators improve interpretability and accuracy of calibration measures.
Support vector machines (SVMs) naturally embody sparseness due to their use of hinge loss functions. However, SVMs can not directly estimate conditional class probabilities. In this paper we propose and study a family of coherence functions, which are convex and differentiable, as surrogates of the hinge function. The …
The majority of traditional classification ru les minimizing the expected probability of error (0-1 loss) are inappropriate if the class probability distributions are ill-defined or impossible to estimate. We argue that in such cases class domains should be used instead of class distributions or densities to construct …
Loss-calibrated EP improves Bayesian decision-making by focusing on utility-sensitive posterior approximations.
problem Bayesian decision-making under asymmetric utility functions.
method Loss-calibrated expectation propagation (Loss-EP) that tilts the posterior towards higher utility decisions.
result Loss-EP can capture useful information for decision-making under asymmetric penalties.
Study on deep neural networks using concentration inequalities and optimal stopping.
problem Understanding the performance and structure of stochastic deep neural networks.
method Introduced concentration inequalities for SDNN outputs and an EC classifier. Determined the optimal number of layers via an optimal stopping procedure.
result Optimal number of layers for SDNNs determined via an optimal stopping procedure.
Maximizes precision and recall at top by thresholding posterior probability.
problem Improving precision and recall at the top in information retrieval and binary classification.
method Thresholding the posterior probability of the positive class.
result Maximizes precision and recall at the top through thresholding.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
MOBA model optimizes error rates for two-class cost-sensitive problems.
problem Cost-sensitive classification with ambiguous labels and varying costs.
method Multi-objective bounded-abstention model using genetic algorithm.
result MOBA achieves lower expected costs with known costs and better trade-offs without.
This monograph deals with adaptive supervised classification, using tools borrowed from statistical mechanics and information theory, stemming from the PACBayesian approach pioneered by David McAllester and applied to a conception of statistical learning theory forged by Vladimir Vapnik. Using convex analysis on the se…
Paper analyzes Gibbs and Langevin Monte Carlo for interpolation regime, showing generalization from low errors.
problem Analyzing Gibbs and Langevin Monte Carlo in overparameterized interpolation regime.
method Data-dependent bounds and stability under approximation with Langevin Monte Carlo.
result Generalization is signaled by small training errors in noisy regime, with bounds stable under approximation.
Multivariate pattern analyses approaches in neuroimaging are fundamentally concerned with investigating the quantity and type of information processed by various regions of the human brain; typically, estimates of classification accuracy are used to quantify information. While a extensive and powerful library of method…
In this paper, we explore degrees of freedom in deep sigmoidal neural networks. We show that the degrees of freedom in these models is related to the expected optimism, which is the expected difference between test error and training error. We provide an efficient Monte-Carlo method to estimate the degrees of freedom f…
Simple linear relationship explains test performance differences in deep networks.
problem Understanding why two deep networks with identical training and architecture have different test performance.
method Showed that cross-entropy loss can lead to drastically different generalization performances for networks with different initialization or corrupted training.
result A linear relationship emerges between training and test losses, revealing the intrinsic problem of measuring test performance with cross-entropy loss.
Probability calibration trees improve accuracy of probability estimates.
problem Improving accuracy and calibration of probability estimates from classifiers.
method Probability calibration trees modify logistic model trees to learn different models in regions of the input space.
result Probability calibration trees outperform isotonic regression and Platt scaling in terms of root mean squared error.
New regularization techniques using mass transportation for better generalization.
problem Mitigating overfitting with scarce data.
method Distributionally robust optimization and worst-case expected loss.
result Generalization bounds and tractable learning problems.
Paper proposes a method to predict MOBA game winners with calibrated confidence.
problem Predicting MOBA game winners with noisy data and uncertain noise.
method A novel confidence-calibration method considering data uncertainty.
result Achieves outstanding expected calibration error (ECE) of 0.57%.
We use partial class memberships in soft classification to model uncertain labelling and mixtures of classes. Partial class memberships are not restricted to predictions, but may also occur in reference labels (ground truth, gold standard diagnosis) for training and validation data. Classifier performance is usually ex…
The problem of estimation error of Expected Shortfall is analyzed, with a view of its introduction as a global regulatory risk measure.
The study quantifies deep learning generalization error using data distribution and network smoothness.
problem Understanding the generalization error in deep learning models.
method Introducing cover complexity (CC) to measure data difficulty and using the inverse of the modulus of continuity to quantify neural network smoothness. A bound for expected accuracy/error is derived considering both CC and neural network smoothness.
result The expected error of trained neural networks scales with the square root of the number of classes and has a linear relationship with respect to the cover complexity.
Develops NPMC method for noisy labels, improving multiclass classification accuracy.
problem Asymmetric misclassification costs and label noise in multiclass classification.
method Empirical likelihood approach using exponential tilting density ratio model.
result Root n consistent and asymptotically normal estimators for clean labels and noise mechanism.
Paper analyzes risk bounds for in-context learning in multiclass classification.
problem Risk bounds for in-context learning in multiclass classification.
method Formalizes tasks as sequences of labeled examples and queries, estimates conditional class probabilities, establishes oracle inequality for KL divergence.
result ICL achieves minimax optimal rate for conditional probability estimation.
We propose a streaming algorithm for the binary classification of data based on crowdsourcing. The algorithm learns the competence of each labeller by comparing her labels to those of other labellers on the same tasks and uses this information to minimize the prediction error rate on each task. We provide performance g…
To train good supervised and semi-supervised object classifiers, it is critical that we not waste the time of the human experts who are providing the training labels. Existing active learning strategies can have uneven performance, being efficient on some datasets but wasteful on others, or inconsistent just between ru…
We introduce the implicitly constrained least squares (ICLS) classifier, a novel semi-supervised version of the least squares classifier. This classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-s…
Novel analysis improves weighted majority vote in multiclass classification.
problem Improving the performance of weighted majority vote in multiclass classification.
method Analyzes expected risk of weighted majority vote, considering prediction correlations and provides a bound for efficient minimization.
result Minimization of the new bound typically does not degrade the test error of the ensemble.
The contour map of estimation error of Expected Shortfall (ES) is constructed. It allows one to quantitatively determine the sample size (the length of the time series) required by the optimization under ES of large institutional portfolios for a given size of the portfolio, at a given confidence level and a given esti…
Theory for soft-margin classifiers on object manifolds.
problem Classifying object manifolds with variability.
method Mean-field theory of soft-margin classifiers applied to object manifolds.
result Prediction of classification errors and their dependence on regularization.
Bayes Error Rate estimators are evaluated for accuracy and sample requirements.
problem Evaluating the accuracy and sample requirements of Bayes Error Rate estimators.
method Monte Carlo simulations with synthetic data and real-world scenarios.
result k-Nearest Neighbor (kNN) is the most accurate non-parametric estimator.
This work bounds classification error in machine learning for low Bayes error conditions.
problem Understanding the error mismatch between Bayes error and model-based classification error.
method Applying classification error bounds to study the relationship with Kullback-Leibler divergence and proposing a linear approximation for low Bayes error conditions.
result A linear approximation of the classification error bound for low Bayes error conditions is proposed.
Improved classification with costly features using deep reinforcement learning.
problem Optimizing classification error with limited and costly feature acquisition.
method Revisited Q-learning approach with neural network approximation for sequential feature requests and classification decisions.
result Deep reinforcement learning approach comparable to state-of-the-art algorithms, robust across datasets.
This paper rethinks confidence calibration under covariate shifts.
problem Calibration methods struggle with covariate shifts and unstable importance weighting.
method Derives Expectation consistency condition and proposes Expectation consistency loss (ECL).
result ECL loss is compatible with various types of calibration and has the same sample complexity as ECE.
Neural network accuracy improves with denser training samples.
problem Improving neural network accuracy on unseen test samples.
method Bounding empirical training error smoothed across activation regions and using it to discard high-risk test samples.
result Discarding high-risk test samples based on error bounds improves prediction accuracy by up to 20%.
Method minimizes electricity procurement cost based on demand prediction errors.
problem Minimizing electricity procurement cost in spot markets.
method Formulate method to minimize procurement cost over two parameters.
result Minimizes total electricity cost with known unit prices and prediction errors.
For the problem of multi-class linear classification and feature selection, we propose approximate message passing approaches to sparse multinomial logistic regression (MLR). First, we propose two algorithms based on the Hybrid Generalized Approximate Message Passing (HyGAMP) framework: one finds the maximum a posterio…
Study evaluates quality of uncertainty estimates for neural networks.
problem Lack of principled assessment methods for evaluating uncertainty quality in deep learning.
method Statistical methods of frequentist interval coverage, interval width, and expected calibration error.
result Different UQ methods produce markedly different quality uncertainty estimates.
For the problem of binary linear classification and feature selection, we propose algorithmic approaches to classifier design based on the generalized approximate message passing (GAMP) algorithm, recently proposed in the context of compressive sensing. We are particularly motivated by problems where the number of feat…
We show how to compute the Bayes error-rate for speaker verifiers.
problem How many errors does a speaker verifier make in a hundred trials?
method We compute the Bayes error-rate using calibrated likelihood ratios and user-supplied prior probabilities.
result The Bayes error-rate is upper bounded by the minimum of EER, P, and 1-P.
The number of trees T in the random forest (RF) algorithm for supervised learning has to be set by the user. It is controversial whether T should simply be set to the largest computationally manageable value or whether a smaller T may in some cases be better. While the principle underlying bagging is that "more trees a…