Unified binary and multiclass margin-based classification methods.
problem No consensus on multiclass loss functions analogous to binary margin loss.
method Showed multiclass loss functions can be expressed in relative margin form.
result Extended classification-calibration result to multiclass.
Classification is the most important process in data analysis. However, due to the inherent non-convex and non-smooth structure of the zero-one loss function of the classification model, various convex surrogate loss functions such as hinge loss, squared hinge loss, logistic loss, and exponential loss are introduced. T…
Establishes a condition for multiclass classification-calibration of Gamma-Phi losses.
problem Ensuring classification-calibration of multiclass Gamma-Phi losses.
method Develops a general sufficient condition for classification-calibration of Gamma-Phi losses.
result Proves the first family of nonconvex multiclass surrogate losses for which classification-calibration has been fully justified.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
We present α α α -loss, α ∈ [ 1 , ∞ ] α\in [1,\infty] α ∈ [ 1 , ∞ ] , a tunable loss function for binary classification that bridges log-loss ( α = 1 α=1 α = 1 ) and 0 0 0 - 1 1 1 loss ( α = ∞ α= \infty α = ∞ ). We prove that α α α -loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal y…
Proposes sigmoidF1 loss for multilabel classification, improving performance metrics.
problem Lack of smooth, tractable loss functions for multilabel classification.
method Introduces sigmoidF1, a smooth F1 score surrogate loss function.
result sigmoidF1 outperforms other loss functions on various datasets and metrics.
Paper explores connections between loss functions and consistency in binary classification and regression.
problem Consistency in binary classification and regression applications.
method Characterization of conformable loss functions and derivation of a new Huber-type loss function.
result Margin-based loss functions are equivalent to loss functions of squared standardized logistic regression residuals.
Develops gradient boosting for multi-label classification.
problem Lack of customizable learning algorithms for multi-label classification.
method Generalizes gradient boosting to multi-output problems and proposes an algorithm for learning multi-label classification rules.
result Ability to minimize both decomposable and non-decomposable loss functions.
Study proposes a differentiable surrogate loss function for optimizing F β F_β F β score in binary classification with imbalanced data.
problem Non-differentiability of F β F_β F β score makes it unsuitable for optimization by gradient-based learning. method Investigated relationship between F β F_β F β score and loss functions, proposed a differentiable surrogate loss function. result Gradient paths of the proposed surrogate F β F_β F β loss function approximate the gradient paths of the F β F_β F β score. We present surrogate regret bounds for arbitrary surrogate losses in the context of binary classification with label-dependent costs. Such bounds relate a classifier's risk, assessed with respect to a surrogate loss, to its cost-sensitive classification risk. Two approaches to surrogate regret bounds are developed. The…
This work proposes the Bregman-Tweedie classification model and analyzes the domain structure of the extended exponential function, an extension of the classic generalized exponential function with additional scaling parameter, and related high-level mathematical structures, such as the Bregman-Tweedie loss function an…
Study on top- k k k classification with new loss functions and algorithms.
problem Improving multi-class classification accuracy and cardinality trade-off.
method Introducing cardinality-aware loss functions and deriving their consistency bounds.
result New cardinality-aware algorithms for top- k k k classification. Conventional techniques for supervised classification constrain the classification rules considered and use surrogate losses for classification 0-1 loss. Favored families of classification rules are those that enjoy parametric representations suitable for surrogate loss minimization, and low complexity properties suita…
This research improves PAC-Bayesian bounds for classification tasks using convexified loss.
problem Deriving generalization bounds for classification tasks with non-convex loss functions.
method Shift focus to misclassification excess risk bounds for PAC-Bayesian classification using convex surrogate loss and leveraging PAC-Bayesian relative bounds in expectation.
result Improved PAC-Bayesian bounds for classification tasks with convex surrogate loss.
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
problem Training two-layer networks for binary classification.
method Gradient descent with logistic loss applied to two-layer networks.
result Gradient descent can drive training loss to zero under certain conditions.
MRCs minimize worst-case expected 0-1 loss and provide performance guarantees.
problem Minimizing expected 0-1 loss in classification.
method Minimizes worst-case expected 0-1 loss over uncertainty sets defined by linear constraints.
result Achieves efficient learning and generalization with performance guarantees.
We introduce a tunable loss function called α α α -loss, parameterized by α ∈ ( 0 , ∞ ] α\in (0,\infty] α ∈ ( 0 , ∞ ] , which interpolates between the exponential loss ( α = 1 / 2 α= 1/2 α = 1/2 ), the log-loss ( α = 1 α= 1 α = 1 ), and the 0-1 loss ( α = ∞ α= \infty α = ∞ ), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $…
This paper introduces new loss functions for balanced multi-class classification.
problem Balancing class imbalance in multi-class classification.
method Introduces two new surrogate loss families: GLA and GCA.
result GCA losses offer stronger theoretical guarantees in imbalanced settings.
The paper studies consistency of surrogate loss procedures under constrained classifiers.
problem Consistency of surrogate loss approaches under constrained classifiers without correct specification.
method The paper develops theoretical results and hinge loss based procedures for a constrained classification problem.
result Hinge losses are the only surrogate losses that preserve consistency in second-best scenarios.
Gaptron algorithm reduces mistakes in online multiclass classification.
problem Online multiclass classification with limited information.
method Randomized first-order algorithm exploiting the gap between zero-one loss and surrogate losses.
result First linear time algorithm with O ( K T ) O(K\sqrt{T}) O ( K T ) expected regret. Focal loss improves classification but not class-posterior probability estimation.
problem Improving class-posterior probability estimation from focal loss.
method Proved classification-calibration and derived a transformation to recover true class-posterior probabilities.
result A transformation of the confidence score from focal loss minimization allows recovery of true class-posterior probabilities.
Paper establishes a universal growth rate for smooth surrogate losses in classification.
problem Analyzing growth rates of consistency bounds for various surrogate losses.
method Proves square-root growth rate for smooth margin-based losses; extends to multi-class classification.
result Demonstrates a universal square-root growth rate for smooth comp-sum and constrained losses.
The paper sets lower bounds for adversarial robustness in multiclass classification.
problem Adversarial robustness in multiclass classification with arbitrary loss functions.
method Dual and barycentric reformulations for robust risk minimization.
result Sharp lower bounds for adversarial risks are computed efficiently.
Paper develops proper, lower-bounded losses for weakly supervised classification.
problem Weakly supervised classification with corrupted labels.
method Representation theorem for proper losses, derived condition for lower-boundedness, generalized logit squeezing.
result Proper and lower-bounded losses for weak-label learning.
New DAM method improves AUC scores in medical image classification.
problem Maximizing AUC in large-scale medical image classification.
method Proposes AUC margin loss for robust optimization, conducts extensive empirical studies.
result Improves performance on four medical image classification tasks, achieving 1st place on Stanford CheXpert.
In classification, the de facto method for aggregating individual losses is the average loss. When the actual metric of interest is 0-1 loss, it is common to minimize the average surrogate loss for some well-behaved (e.g. convex) surrogate. Recently, several other aggregate losses such as the maximal loss and average t…
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from co…
EnsLoss combines multiple loss functions to prevent overfitting in classification.
problem Preventing overfitting in classification models.
method EnsLoss is an ensemble method that combines loss functions, ensuring calibration and consistency.
result EnsLoss improves classification accuracy compared to fixed loss methods.
Paper introduces MRCs that minimize worst-case 0-1 loss, providing tight performance guarantees.
problem Minimizing worst-case 0-1 loss in classification.
method MRCs that minimize worst-case 0-1 loss with uncertainty sets of distributions.
result MRCs provide tight performance guarantees and are strongly universally consistent.
Classification and regression tasks in overparameterized models show different generalization properties.
problem Comparing classification and regression in overparameterized models.
method Comparison of least-squares minimum-norm interpolation and hard-margin SVM using different loss functions.
result Interpolating solutions generalize well with 0-1 loss but not with square loss.
Paper introduces a new loss function for deep imbalanced classification.
problem Class ambiguity and imbalance in large datasets.
method Stochastic top-K hinge loss based on smoothed top-K operator.
result Our loss function significantly outperforms other baseline loss functions in imbalanced datasets.
Reduces bounded loss learning to binary classification.
problem Universal consistency of non-i.i.d. processes with bounded loss.
method Constructive reduction to binary classification.
result Any bounded loss output setting can be reduced to binary classification.
Paper unifies bias and variance models for classification.
problem Different frameworks for bias and variance in classification.
method Unified Tumer & Ghosh and James approaches.
result Closed form relationships between 0/1 loss and squared error loss.
GE2E-AC improves accent classification by focusing on accent embeddings.
problem Training models to predict accent type can lead to learning irrelevant features.
method GE2E-AC trains models to extract accent embeddings, making them closer for the same accent class.
result GE2E-AC outperforms baseline models trained with conventional loss.
Structured entropy improves classification performance on structured targets.
problem Cross-entropy loss fails to account for target variable structure.
method Proposes structured entropy, a generalization of entropy using random partitions.
result Structured cross-entropy loss yields better results on classification problems with known structure.
Study improves top-k set prediction with low cardinality.
problem Improving top-k set prediction accuracy with low cardinality.
method Introduces new target loss function and surrogate losses.
result Demonstrates effectiveness of cardinality-aware algorithms.
Unhinged loss minimization fails to improve classifier accuracy for simple data.
problem Accuracy of classifiers minimizing the unhinged loss.
method Minimizing the unhinged loss function.
result Minimizing the unhinged loss yields classifiers with accuracy no better than random guessing for simple data.
This work examines uncertainty sampling in binary classification using equivalent loss.
problem Lack of consensus on proper uncertainty definition and theoretical guarantees for active learning.
method Systematically examines uncertainty sampling via equivalent loss, proving its optimality.
result Established that uncertainty sampling optimizes against equivalent loss, providing theoretical guarantees.
New loss functions improve extreme classification with missing labels.
problem Large number of infrequent labels and missing labels in XMC.
method Derive unbiased loss functions for XMC, incorporating them into existing algorithms.
result Significant improvement in extreme classification performance (up to 20%) over existing methods.
We propose a robust adversarial prediction framework for general multiclass classification. Our method seeks predictive distributions that robustly optimize non-convex and non-continuous multiclass loss metrics against the worst-case conditional label distributions (the adversarial distributions) that (approximately) m…
Novel loss function improves few-shot classification performance.
problem Few-shot classification struggles with embedding space for new classes.
method Proposes self-compacting softmax loss (SSL) to prevent embedding space fullness.
result SSL leads to significant improvement in state-of-the-art performance.
Paper proposes ARB-Loss to improve classification precision in imbalanced datasets.
problem Improving classification precision on minor classes in imbalanced datasets.
method Introduces Attraction-Repulsion-Balanced Loss (ARB-Loss) to balance gradients across different classes.
result ARB-Loss achieves state-of-the-art performance with one-stage training.
Boosted CVaR Classification improves tail performance in classification tasks.
problem Maximizing tail performance in classification tasks.
method Proposed Boosted CVaR Classification framework using randomized classifiers and LPBoost algorithm.
result Minimizing CVaR loss over randomized classifiers leads to better tail performance.
Paper explains learning property of logistic and softmax losses for balanced and imbalanced class data.
problem Understanding and optimizing loss functions for deep neural networks with class imbalances.
method Analyzing necessary conditions for convergence of logistic and softmax losses in CNNs.
result Proposes a novel reweighted logistic loss function that improves performance over softmax loss.
This work extends implicit bias analysis to multiclass classification using a new loss framework.
problem The implicit bias of gradient descent on multiclass data without explicit regularization.
method Employing the PERM framework to introduce a multiclass extension of the exponential tail property.
result Extended implicit bias result to multiclass classification using a new loss framework.
Symmetric losses improve classifier robustness from corrupted labels.
problem Improving classifier performance from corrupted labels.
method Symmetric losses that satisfy a certain condition.
result Symmetric losses enhance robust classification from corrupted labels.
A novel method for classification with rejection using ensemble of cost-sensitive classifiers.
problem Avoid risky misclassification in error-critical applications.
method Learning an ensemble of cost-sensitive classifiers.
result Improved classification accuracy and flexibility in loss selection.
Paper introduces negative margin loss for better few-shot classification accuracy.
problem Improving few-shot classification accuracy with metric learning.
method Introduces negative margin loss and analyzes its impact on feature discriminability.
result Negative margin loss outperforms regular softmax loss on few-shot classification benchmarks.