New loss function calibrates WW-hinge loss for multiclass SVM.
problem WW-hinge loss not calibrated with 0-1 loss.
method Introduced ordered partition loss and proved WW-hinge loss is calibrated.
result WW-hinge loss is calibrated with ordered partition loss.
Efficient algorithms for large-scale multiclass classification with linear classifiers.
problem Training ℓ 1 \ell_1 ℓ 1 -regularized linear classifiers with high dimensionality and many classes. method Combines quasi-bilinear objective, stochastic mirror descent, and non-uniform sampling.
result Proposes a sublinear algorithm for multiclass hinge loss.
We present an efficient second-order algorithm with O ~ ( 1 η T ) \tilde{O}(\frac{1}η\sqrt{T}) O ~ ( η 1 T ) regret for the bandit online multiclass problem. The regret bound holds simultaneously with respect to a family of loss functions parameterized by η η η , for a range of η η η restricted by the norm of the competitor. The family of loss funct…
Gaptron algorithm reduces mistakes in online multiclass classification.
problem Online multiclass classification with limited information.
method Randomized first-order algorithm exploiting the gap between zero-one loss and surrogate losses.
result First linear time algorithm with O ( K T ) O(K\sqrt{T}) O ( K T ) expected regret. Unified binary and multiclass margin-based classification methods.
problem No consensus on multiclass loss functions analogous to binary margin loss.
method Showed multiclass loss functions can be expressed in relative margin form.
result Extended classification-calibration result to multiclass.
We extend the well-known BFGS quasi-Newton method and its memory-limited variant LBFGS to the optimization of nonsmooth convex objectives. This is done in a rigorous fashion by generalizing three components of BFGS to subdifferentials: the local quadratic model, the identification of a descent direction, and the Wolfe …
Establishes a condition for multiclass classification-calibration of Gamma-Phi losses.
problem Ensuring classification-calibration of multiclass Gamma-Phi losses.
method Develops a general sufficient condition for classification-calibration of Gamma-Phi losses.
result Proves the first family of nonconvex multiclass surrogate losses for which classification-calibration has been fully justified.
Gradient descent converges to max-margin solution for hinge loss.
problem Applying gradient descent to the hinge loss for linear classifiers.
method Homotopic gradient descent applied to the hinge loss.
result Explicit convergence rates to max-margin solution for separable data.
Logitron combines Perceptron and logistic loss for improved classification.
problem Non-convex and non-smooth zero-one loss function in classification models.
method Introduces a Perceptron-augmented convex classification framework with an extended logistic loss function.
result Hinge-Logitron outperforms logistic regression and SVM in classification accuracy.
We study consistency properties of surrogate loss functions for general multiclass learning problems, defined by a general multiclass loss matrix. We extend the notion of classification calibration, which has been studied for binary and multiclass 0-1 classification problems (and for certain other specific learning pro…
We propose a robust adversarial prediction framework for general multiclass classification. Our method seeks predictive distributions that robustly optimize non-convex and non-continuous multiclass loss metrics against the worst-case conditional label distributions (the adversarial distributions) that (approximately) m…
New loss function improves convergence rate for neural networks.
problem Improving convergence speed of neural networks for classification tasks.
method Proposes a modified hinge loss function with gradients to critical points.
result Margin converges to max-margin at O ( 1 / t ) \mathcal{O}(1/t) O ( 1/ t ) rate, faster than exponential losses. This work extends implicit bias analysis to multiclass classification using a new loss framework.
problem The implicit bias of gradient descent on multiclass data without explicit regularization.
method Employing the PERM framework to introduce a multiclass extension of the exponential tail property.
result Extended implicit bias result to multiclass classification using a new loss framework.
Paper studies multiclass classifiers from binary classifiers, proving methods and demonstrating advantages.
problem Constructing efficient multiclass classifiers from binary ones.
method Two methods: one vs. all and hierarchical classification, with a new leverage-hierarchical method introduced.
result Proves upper bounds and exact formulas for multiclass regret in terms of binary regrets.
Improves GANs by incorporating class information with a multi-hinge loss.
problem Improving GANs to better respect class information.
method Proposes a multi-class generalization of the Hinge loss for GANs.
result Improves Inception Scores and Frechet Inception Distance on Imagenet.
A new method learns proper multiclass losses and probabilities.
problem Learning proper multiclass losses for complex classification tasks.
method Extends monotonicity to multiclass problems using convex functions.
result Consistently outperforms natural multiclass baseline on up to 1,000 class datasets.
Characterizes learnability of forgiving 0-1 loss functions in multiclass settings.
problem Understanding when multiclass learning with forgiving 0-1 loss functions is possible.
method Introduces a new combinatorial dimension based on Natarajan Dimension to determine learnability.
result A hypothesis class is learnable if and only if the Generalized Natarajan Dimension is finite.
The paper sets lower bounds for adversarial robustness in multiclass classification.
problem Adversarial robustness in multiclass classification with arbitrary loss functions.
method Dual and barycentric reformulations for robust risk minimization.
result Sharp lower bounds for adversarial risks are computed efficiently.
The paper analyzes top-k classification and proposes consistent loss functions.
problem Understanding consistency of top-k classification in challenging tasks.
method Theoretical analysis, defining top-k calibration, proposing new loss functions.
result Proposes a new consistent hinge loss and a top-k calibrated convex loss.
Investigates multiclass classification with rejection, achieving state-of-the-art performance and deriving calibration conditions.
problem Multiclass classification with rejection, where a classifier can choose not to predict.
method Two approaches: simultaneous training of classifier and rejector, and confidence scores with rejection criteria.
result Calibration is hard for general loss functions in multiclass case, but achievable with specific rejection criteria.
We consider composite loss functions for multiclass prediction comprising a proper (i.e., Fisher-consistent) loss over probability distributions and an inverse link function. We establish conditions for their (strong) convexity and explore the implications. We also show how the separation of concerns afforded by using …
The paper studies consistency of surrogate loss procedures under constrained classifiers.
problem Consistency of surrogate loss approaches under constrained classifiers without correct specification.
method The paper develops theoretical results and hinge loss based procedures for a constrained classification problem.
result Hinge losses are the only surrogate losses that preserve consistency in second-best scenarios.
We consider the problem of n n n -class classification ( n ≥ 2 n\geq 2 n ≥ 2 ), where the classifier can choose to abstain from making predictions at a given cost, say, a factor α α α of the cost of misclassification. Designing consistent algorithms for such n n n -class classification problems with a `reject option' is the main goal of t…
Top-k error is currently a popular performance measure on large scale image classification benchmarks such as ImageNet and Places. Despite its wide acceptance, our understanding of this metric is limited as most of the previous research is focused on its special case, the top-1 error. In this work, we explore two direc…
New approach uses hinge loss for iterative regularization in classification.
problem Improving classification accuracy through regularization.
method Develops an iterative regularization approach based on hinge loss.
result Proves convergence and rates of convergence for classification.
New robust metric learning method improves performance in noisy data.
problem Label noise and outliers in training data degrade metric learning performance.
method Rescaled Hinge loss function and HQ algorithm.
result New method outperforms state-of-the-art methods in noisy data.
Support vector machines (SVMs) naturally embody sparseness due to their use of hinge loss functions. However, SVMs can not directly estimate conditional class probabilities. In this paper we propose and study a family of coherence functions, which are convex and differentiable, as surrogates of the hinge function. The …
Learning with non-modular losses is an important problem when sets of predictions are made simultaneously. The main tools for constructing convex surrogate loss functions for set prediction are margin rescaling and slack rescaling. In this work, we show that these strategies lead to tight convex surrogates iff the unde…
AUC (area under ROC curve) is an important evaluation criterion, which has been popularly used in many learning tasks such as class-imbalance learning, cost-sensitive learning, learning to rank, etc. Many learning approaches try to optimize AUC, while owing to the non-convexity and discontinuousness of AUC, almost all …
New loss functions improve extreme classification with missing labels.
problem Large number of infrequent labels and missing labels in XMC.
method Derive unbiased loss functions for XMC, incorporating them into existing algorithms.
result Significant improvement in extreme classification performance (up to 20%) over existing methods.
Paper proposes a boosting method with fast learning rates and early stopping.
problem Missing theoretical guarantees for boosting methods in binary classification.
method Fully-corrective gradient boosting with squared hinge loss and ADMM algorithm.
result Derives fast learning rates of O ( ( m / log m ) − 1 / 4 ) {\cal O}((m/\log m)^{-1/4}) O (( m / log m ) − 1/4 ) and O ( ( m / log m ) − 1 / 2 ) {\cal O}((m/\log m)^{-1/2}) O (( m / log m ) − 1/2 ) . Proposes hinge-Wasserstein to improve uncertainty estimation in regression tasks.
problem Estimating multimodal aleatoric uncertainty in regression tasks from images.
method Regression-by-classification paradigm with hinge-Wasserstein loss.
result Hinge-Wasserstein loss improves uncertainty estimation on challenging tasks.
Paper introduces a new loss function for deep imbalanced classification.
problem Class ambiguity and imbalance in large datasets.
method Stochastic top-K hinge loss based on smoothed top-K operator.
result Our loss function significantly outperforms other baseline loss functions in imbalanced datasets.
The Nyström method improves learning efficiency for convex losses.
problem Improving computational efficiency in empirical risk minimization.
method Using random subspaces to approximate hypothesis spaces in convex loss functions.
result Computational gains can be achieved without sacrificing learning performance for general convex Lipschitz losses.
JoVA combines two VAEs to learn user and item representations for better recommendation.
problem Collaborative filtering with implicit feedback.
method Joint Variational Autoencoders (JoVA) with a hinge-based pairwise loss function (JoVA-Hinge).
result JoVA-Hinge outperforms state-of-the-art methods in top-k recommendation.
The paper explores symmetric losses for better learning from corrupted labels.
problem Learning from corrupted labels with balanced error rate or AUC maximization.
method Proves theoretical properties of symmetric losses and proposes a convex barrier hinge loss.
result Symmetric losses are advantageous in BER minimization and AUC maximization from corrupted labels.
A robust multiclass SVM tackles imbalanced data uncertainty.
problem Imbalanced multiclass classification with uncertain labels.
method Wasserstein distributionally robust optimization for multiclass SVM.
result Our model outperforms state-of-the-art OVA models in imbalanced settings.
A new procedure for learning cost-sensitive SVM(CS-SVM) classifiers is proposed. The SVM hinge loss is extended to the cost sensitive setting, and the CS-SVM is derived as the minimizer of the associated risk. The extension of the hinge loss draws on recent connections between risk minimization and probability elicitat…
Introduces Soft-SVM for binary classification bridging logistic and SVM.
problem Data separability issues in binary classification.
method Soft-SVM regression using convex relaxation of hinge loss with softness and class-separation parameters.
result Soft-SVM performs well in classification and prediction errors.
Study efficient learning of halfspaces with constant noise tolerance.
problem Learning halfspaces in the presence of both instance and label corruption.
method Develops an algorithm to minimize reweighted hinge loss for robustness.
result Achieves constant noise tolerance for halfspace learning.
New research shows SVM and related methods can overfit without harm in multiclass classification.
problem Understanding benign overfitting in multiclass classification.
method Analyzing three training algorithms: ERM with cross-entropy, least-squares, and one-vs-all SVM.
result All three algorithms can lead to classifiers that interpolate training data and have equal accuracy under high overparameterization.
Study extends binary omniprediction to multiclass setting with improved sample complexity.
problem Suboptimality bounds for each loss function against infinite comparator family in multiclass prediction.
method Design of a framework for solving Blackwell approachability problems with coupled actions.
result Sample complexity of ≈ ε − ( k + 1 ) \approx \varepsilon^{-(k+1)} ≈ ε − ( k + 1 ) for ε \varepsilon ε -omniprediction in a k k k -class problem. New framework enhances neural network robustness against adversarial attacks.
problem Vulnerability of deep neural networks to small perturbations.
method Integrates Lipschitz constraint using optimal transport and hinge regularization.
result Proposes a new loss function that certifies adversarial robustness.
Optimal multiclass U-calibration error found to be Θ(√KT).
problem Online multiclass U-calibration with low regret for all bounded proper losses.
method Follow-the-Perturbed-Leader algorithm and lower bound construction.
result Optimal U-calibration error is Θ(√KT).
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…
A new method for multiclass calibration using vector quantization.
problem Challenges in multiclass calibration, especially in high-stakes settings.
method Compositional approach via Vector Quantization (VQ) to learn region-specific calibration maps.
result Significant improvements in local calibration with competitive global calibration and predictive performance.
The paper revisits discriminative vs. generative classifiers, showing naive Bayes requires fewer samples.
problem Comparing discriminative and generative classifiers in multiclass settings.
method Theoretical analysis and simulations of naive Bayes vs. logistic regression.
result Multiclass naive Bayes requires fewer samples to approach asymptotic error compared to logistic regression.
Classification and regression tasks in overparameterized models show different generalization properties.
problem Comparing classification and regression in overparameterized models.
method Comparison of least-squares minimum-norm interpolation and hard-margin SVM using different loss functions.
result Interpolating solutions generalize well with 0-1 loss but not with square loss.