Unified binary and multiclass margin-based classification methods.
problem No consensus on multiclass loss functions analogous to binary margin loss.
method Showed multiclass loss functions can be expressed in relative margin form.
result Extended classification-calibration result to multiclass.
Paper introduces negative margin loss for better few-shot classification accuracy.
problem Improving few-shot classification accuracy with metric learning.
method Introduces negative margin loss and analyzes its impact on feature discriminability.
result Negative margin loss outperforms regular softmax loss on few-shot classification benchmarks.
Study shows exponential error reduction in multiclass classification without bias-variance trade-off.
problem Multiclass classification with margin conditions.
method Analysis of classification error under hard-margin conditions.
result Exponential decrease in classification error without bias-variance trade-off.
Near-Exponential Convergence Rates for kNN Classification
problem Convergence rates for kNN classification
method Introducing Boltzmann margin
result First near-exponential convergence rates for kNN classification
The paper improves SVM margin-based generalization bounds.
problem Improving generalization bounds for SVMs.
method Revisiting and improving classic generalization bounds in terms of margins, complementing with a nearly matching lower bound.
result Almost settles the generalization performance of SVMs in terms of margins.
Theory for soft-margin classifiers on object manifolds.
problem Classifying object manifolds with variability.
method Mean-field theory of soft-margin classifiers applied to object manifolds.
result Prediction of classification errors and their dependence on regularization.
Study on neuron dynamics for XOR classification with zero-margin.
problem Understanding neural network training dynamics in zero-margin classification problems.
method Analysis of Gaussian XOR problem, focusing on neuron block dynamics and generalization without margin assumptions.
result Neurons cluster into four directions and block-level signals evolve coherently, essential for reliable prediction in the Gaussian setting.
New margin-based regularization and selective sampling improve deep neural network performance.
problem Improving deep neural network performance on various classification tasks.
method Multi-margin regularization (MMR) and minimal margin score (MMS) for selective sampling.
result Improved results on multiple classification tasks across domains.
Neural networks can approximate high-dimensional classifiers with ReLU networks under margin conditions.
problem Approximating high-dimensional discontinuous classifiers with neural networks.
method Using ReLU neural networks with three hidden layers, approximating a classifier with a Barron-regular decision boundary.
result High-dimensional discontinuous classifiers can be approximated with a rate of n−1 under strong margin conditions. A new approach to cost-sensitive multiclass classification prioritizes certain classes over others.
problem Cost-sensitive multiclass classification where some classes are more important than others.
method Apportioned margin framework that shifts the decision boundary to prioritize certain classes.
result The method improves the error rate for important classes while reducing overall error.
New insights into deep learning: reducing training data significantly improves performance.
problem Understanding and improving generalization in deep learning models.
method Analyzing the distribution of classification margins and dynamically reducing the training set.
result The area under the curve of the margin distribution is a good measure of generalization.
We provide a new local class-purity theorem for Lipschitz continuous DNN classifiers. In addition, we discuss how to achieve classification margin for training samples. Finally, we describe how to compute margin p-values for test samples.
We present a formulation of deep learning that aims at producing a large margin classifier. The notion of margin, minimum distance to a decision boundary, has served as the foundation of several theoretically profound and empirically successful results for both classification and regression tasks. However, most large m…
Study shows how over-parameterized classifiers can still perform well on noisy data.
problem Understanding how maximum margin classifiers perform in over-parameterized settings with noisy data.
method Analyzes maximum margin classifiers on sub-Gaussian mixtures, providing risk bounds.
result Characterizes conditions for 'benign overfitting' in linear classification problems.
New method accelerates large margin metric learning for nearest neighbor classification.
problem Efficiently learning metrics for nearest neighbor classification.
method Triplet mining and stratified sampling for large margin metric learning.
result Improved efficiency and scalability of optimization.
Random projections have been applied in many machine learning algorithms. However, whether margin is preserved after random projection is non-trivial and not well studied. In this paper we analyse margin distortion after random projection, and give the conditions of margin preservation for binary classification problem…
MCD reformulates conditional density estimation into binary classification.
problem Conditional density estimation in statistical and machine learning.
method Marginal Contrastive Discrimination, reformulating into marginal and ratio density functions for binary classification.
result Significantly outperforms existing methods on most density models and regression datasets.
Study shows how transformers classify symbols without naming them, proving a margin-versus-collision criterion.
problem How transformers classify symbols without naming them.
method Logistic classification analysis of transformer-kernel regime, colored collision graph.
result Decomposes learned predictor into ideal template-level classifier and finite-sample perturbation.
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
Max-margin learning is a powerful approach to building classifiers and structured output predictors. Recent work on max-margin supervised topic models has successfully integrated it with Bayesian topic models to discover discriminative latent semantic structures and make accurate predictions for unseen testing data. Ho…
New loss function improves convergence rate for neural networks.
problem Improving convergence speed of neural networks for classification tasks.
method Proposes a modified hinge loss function with gradients to critical points.
result Margin converges to max-margin at O(1/t) rate, faster than exponential losses. Algorithm improves SVM classification in non-Euclidean spaces.
problem Limitations of traditional SVM in non-Euclidean spaces.
method Covariance-adjusted SVM using Cholesky Decomposition.
result Cholesky-SVM outperforms traditional SVM in non-Euclidean spaces.
MARVEL curbs memorization of noisy labels in deep nets.
problem Noisy labels degrade deep net performance.
method MARVEL tracks classification margins to identify and abandon noisy instances.
result MARVEL outperforms baselines on noisy datasets.
Paper corrects Max-Margin loss for multi-label tasks.
problem Max-Margin loss inconsistency in multi-label classification.
method Introduced Restricted-Max-Margin loss.
result Consistent loss for multi-label tasks under milder conditions.
New DAM method improves AUC scores in medical image classification.
problem Maximizing AUC in large-scale medical image classification.
method Proposes AUC margin loss for robust optimization, conducts extensive empirical studies.
result Improves performance on four medical image classification tasks, achieving 1st place on Stanford CheXpert.
We consider a problem of risk estimation for large-margin multi-class classifiers. We propose a novel risk bound for the multi-class classification problem. The bound involves the marginal distribution of the classifier and the Rademacher complexity of the hypothesis class. We prove that our bound is tight in the numbe…
Margin-based classifiers have been popular in both machine learning and statistics for classification problems. Since a large number of classifiers are available, one natural question is which type of classifiers should be used given a particular classification task. We answer this question by investigating the asympto…
Max-min margin Markov networks improve consistency in structured prediction.
problem Statistical inconsistency in max-margin methods for structured prediction.
method Defining a max-min margin formulation to overcome statistical inconsistency.
result Proves consistency and provides an explicit algorithm with finite sample generalization bounds.
In this work, we study a new approach to optimizing the margin distribution realized by binary classifiers. The classical approach to this problem is simply maximization of the expected margin, while more recent proposals consider simultaneous variance control and proxy objectives based on robust location estimates, in…
Paper improves DP-ERM for binary linear classification with large-margin subsets.
problem Differentially private binary linear classification with large-margin subsets.
method Efficient (ε,δ)-DP algorithm with empirical zero-one risk bound. result Improved empirical zero-one risk bound for binary linear classification.
We obtain a tight distribution-specific characterization of the sample complexity of large-margin classification with L_2 regularization: We introduce the γ-adapted-dimension, which is a simple function of the spectrum of a distribution's covariance matrix, and show distribution-specific upper and lower bounds on the s…
Paper explores connections between loss functions and consistency in binary classification and regression.
problem Consistency in binary classification and regression applications.
method Characterization of conformable loss functions and derivation of a new Huber-type loss function.
result Margin-based loss functions are equivalent to loss functions of squared standardized logistic regression residuals.
Enhanced consistency bounds derived for classification under a new noise condition.
problem Enhanced consistency bounds for classification under a new noise condition.
method Model Margin Noise (MM noise) assumption, derived enhanced H-consistency bounds.
result Enhanced H-consistency bounds under MM noise condition, interpolates between linear and square-root regimes.
Study minimax rates for binary classifier estimation with margin conditions.
problem Estimating binary classifiers with geometric margin conditions.
method Derive lower bounds for worst-case learning rates over various function classes.
result Identify optimal rates close to O(n−1) for different function classes. Active learning can't improve over passive in certain settings.
problem Active learning vs. passive learning in nonparametric settings.
method Analyzing margin conditions and their effects on active learning performance.
result Nuances in margin conditions determine whether active learning can outperform passive learning.
The paper develops distribution-free methods for ordinal classification.
problem Constructing valid prediction sets for ordinal classification problems.
method Leveraging conformal prediction and multiple testing with FWER control.
result The proposed methods achieve satisfactory levels of marginal and class-specific conditional coverages.
Multi-class classification with a very large number of classes, or extreme classification, is a challenging problem from both statistical and computational perspectives. Most of the classical approaches to multi-class classification, including one-vs-rest or multi-class support vector machines, require the exact estima…
SGD converges to critical points of normalized margin in late-stage training for homogeneous neural networks.
problem Analyzing the implicit bias of SGD on homogeneous neural networks.
method Interpreting SGD dynamics as an Euler-like discretization of a conservative field flow associated with the normalized classification margin.
result Normalized SGD iterates converge to the set of critical points of the normalized margin at late-stage training.
Maximizing margins leads to lossless compression of training data.
problem Generalization in supervised learning.
method Information-theoretic interpretation of margin maximization.
result Margin maximization is a form of lossless maximal compression.
Risk bounds for Classification and Regression Trees (CART, Breiman et. al. 1984) classifiers are obtained under a margin condition in the binary supervised classification framework. These risk bounds are obtained conditionally on the construction of the maximal deep binary tree and permit to prove that the linear penal…
The paper analyzes the maximum margin algorithm's performance on noisy data.
problem Analyzing the performance of maximum margin algorithm on noisy data.
method Finite-sample analysis of maximum margin algorithm applied to noisy data.
result The maximum margin algorithm can achieve nearly optimal population risk with sufficient over-parameterization.
Paper proposes a new classifier for hyperbolic spaces using horospherical boundaries.
problem Optimization of large margin classifiers in hyperbolic spaces.
method Horospherical decision boundaries for geodesically convex optimization.
result Geodesically convex optimization leads to globally optimal solutions.
We obtain a tight distribution-specific characterization of the sample complexity of large-margin classification with L2 regularization: We introduce the margin-adapted dimension, which is a simple function of the second order statistics of the data distribution, and show distribution-specific upper and lower bounds on…
In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference is a likelihood-free method as its gradient can be sampled without bias and without requiring any evaluation of eith…
We consider the problem of adaptation to the margin and to complexity in binary classification. We suggest an exponential weighting aggregation scheme. We use this aggregation procedure to construct classifiers which adapt automatically to margin and complexity. Two main examples are worked out in which adaptivity is a…
We consider the problem of online multiclass classification with partial feedback, where an algorithm predicts a class for a new instance in each round and only receives its correctness. Although several methods have been developed for this problem, recent challenging real-world applications require further performance…
Study generalization of voting classifiers using margin-based bounds.
problem Understanding the generalization of ensemble classifiers like voting.
method Proved margin-based generalization bounds using PAC-Bayes theory and Dirichlet posteriors.
result Provided state-of-the-art guarantees on classification tasks.
This paper studies binary classification problem associated with a family of loss functions called large-margin unified machines (LUM), which offers a natural bridge between distribution-based likelihood approaches and margin-based approaches. It also can overcome the so-called data piling issue of support vector machi…