The paper develops consistent algorithms for multiclass classification with a reject option.
problem Designing consistent algorithms for multiclass classification with a reject option.
method The paper introduces and analyzes three consistent algorithms: Crammer-Singer surrogate, one vs all hinge loss, and a new convex surrogate.
result The new convex surrogate is consistent for any α ∈ [0, 1/2] and operates on a lower dimensional space.
Classification and regression tasks in overparameterized models show different generalization properties.
problem Comparing classification and regression in overparameterized models.
method Comparison of least-squares minimum-norm interpolation and hard-margin SVM using different loss functions.
result Interpolating solutions generalize well with 0-1 loss but not with square loss.
Neural networks for binary classification have zero training error at all local minima under certain conditions.
problem Understanding the loss surface of neural networks for binary classification.
method Analyzing single-layered neural networks with smooth hinge loss function, providing conditions for zero training error at all local minima.
result Zero training error at all local minima is achieved under specific conditions (strict convexity of neurons and smooth hinge loss).
Gradient descent converges to max-margin solution for hinge loss.
problem Applying gradient descent to the hinge loss for linear classifiers.
method Homotopic gradient descent applied to the hinge loss.
result Explicit convergence rates to max-margin solution for separable data.
AUC (area under ROC curve) is an important evaluation criterion, which has been popularly used in many learning tasks such as class-imbalance learning, cost-sensitive learning, learning to rank, etc. Many learning approaches try to optimize AUC, while owing to the non-convexity and discontinuousness of AUC, almost all …
New loss function calibrates WW-hinge loss for multiclass SVM.
problem WW-hinge loss not calibrated with 0-1 loss.
method Introduced ordered partition loss and proved WW-hinge loss is calibrated.
result WW-hinge loss is calibrated with ordered partition loss.
Logitron combines Perceptron and logistic loss for improved classification.
problem Non-convex and non-smooth zero-one loss function in classification models.
method Introduces a Perceptron-augmented convex classification framework with an extended logistic loss function.
result Hinge-Logitron outperforms logistic regression and SVM in classification accuracy.
The paper explores symmetric losses for better learning from corrupted labels.
problem Learning from corrupted labels with balanced error rate or AUC maximization.
method Proves theoretical properties of symmetric losses and proposes a convex barrier hinge loss.
result Symmetric losses are advantageous in BER minimization and AUC maximization from corrupted labels.
New loss function improves convergence rate for neural networks.
problem Improving convergence speed of neural networks for classification tasks.
method Proposes a modified hinge loss function with gradients to critical points.
result Margin converges to max-margin at O ( 1 / t ) \mathcal{O}(1/t) O ( 1/ t ) rate, faster than exponential losses. Improves GANs by incorporating class information with a multi-hinge loss.
problem Improving GANs to better respect class information.
method Proposes a multi-class generalization of the Hinge loss for GANs.
result Improves Inception Scores and Frechet Inception Distance on Imagenet.
Deep neural networks converge quickly for classification tasks.
problem Classifying data with smooth decision boundaries, probabilities, or margins.
method Hinge loss and cross-entropy for training; analysis of convergence rates.
result DNNs achieve fast convergence rates under various conditions.
MAGAN improves GANs stability and performance with adaptive hinge loss.
problem Improving stability and performance of GANs.
method Adaptive hinge loss function with estimated margin based on target distribution energy.
result MAGAN converges to global optimum under certain assumptions.
The paper analyzes top-k classification and proposes consistent loss functions.
problem Understanding consistency of top-k classification in challenging tasks.
method Theoretical analysis, defining top-k calibration, proposing new loss functions.
result Proposes a new consistent hinge loss and a top-k calibrated convex loss.
We carefully study how well minimizing convex surrogate loss functions, corresponds to minimizing the misclassification error rate for the problem of binary classification with linear predictors. In particular, we show that amongst all convex surrogate losses, the hinge loss gives essentially the best possible bound, o…
Novel convex surrogate for submodular losses with tractable computation.
problem Learning with non-modular losses for set prediction.
method Proposed Lovász hinge loss function for submodular losses.
result First tractable convex surrogates for submodular losses.
The paper studies consistency of surrogate loss procedures under constrained classifiers.
problem Consistency of surrogate loss approaches under constrained classifiers without correct specification.
method The paper develops theoretical results and hinge loss based procedures for a constrained classification problem.
result Hinge losses are the only surrogate losses that preserve consistency in second-best scenarios.
New approach uses hinge loss for iterative regularization in classification.
problem Improving classification accuracy through regularization.
method Develops an iterative regularization approach based on hinge loss.
result Proves convergence and rates of convergence for classification.
New robust metric learning method improves performance in noisy data.
problem Label noise and outliers in training data degrade metric learning performance.
method Rescaled Hinge loss function and HQ algorithm.
result New method outperforms state-of-the-art methods in noisy data.
Paper explains learning property of logistic and softmax losses for balanced and imbalanced class data.
problem Understanding and optimizing loss functions for deep neural networks with class imbalances.
method Analyzing necessary conditions for convergence of logistic and softmax losses in CNNs.
result Proposes a novel reweighted logistic loss function that improves performance over softmax loss.
Support vector machines (SVMs) naturally embody sparseness due to their use of hinge loss functions. However, SVMs can not directly estimate conditional class probabilities. In this paper we propose and study a family of coherence functions, which are convex and differentiable, as surrogates of the hinge function. The …
New algorithms and bounds for contextual bandits using surrogate losses.
problem Efficiently solving contextual bandit problems with margin-based regret bounds.
method Use of surrogate losses (ramp and hinge) to derive new regret bounds and algorithms.
result Derives new margin-based regret bounds and efficient algorithms for contextual bandits.
New loss functions improve extreme classification with missing labels.
problem Large number of infrequent labels and missing labels in XMC.
method Derive unbiased loss functions for XMC, incorporating them into existing algorithms.
result Significant improvement in extreme classification performance (up to 20%) over existing methods.
Efficient algorithms for large-scale multiclass classification with linear classifiers.
problem Training ℓ 1 \ell_1 ℓ 1 -regularized linear classifiers with high dimensionality and many classes. method Combines quasi-bilinear objective, stochastic mirror descent, and non-uniform sampling.
result Proposes a sublinear algorithm for multiclass hinge loss.
Paper proposes a boosting method with fast learning rates and early stopping.
problem Missing theoretical guarantees for boosting methods in binary classification.
method Fully-corrective gradient boosting with squared hinge loss and ADMM algorithm.
result Derives fast learning rates of O ( ( m / log m ) − 1 / 4 ) {\cal O}((m/\log m)^{-1/4}) O (( m / log m ) − 1/4 ) and O ( ( m / log m ) − 1 / 2 ) {\cal O}((m/\log m)^{-1/2}) O (( m / log m ) − 1/2 ) . Proposes hinge-Wasserstein to improve uncertainty estimation in regression tasks.
problem Estimating multimodal aleatoric uncertainty in regression tasks from images.
method Regression-by-classification paradigm with hinge-Wasserstein loss.
result Hinge-Wasserstein loss improves uncertainty estimation on challenging tasks.
Paper introduces a new loss function for deep imbalanced classification.
problem Class ambiguity and imbalance in large datasets.
method Stochastic top-K hinge loss based on smoothed top-K operator.
result Our loss function significantly outperforms other baseline loss functions in imbalanced datasets.
The Nyström method improves learning efficiency for convex losses.
problem Improving computational efficiency in empirical risk minimization.
method Using random subspaces to approximate hypothesis spaces in convex loss functions.
result Computational gains can be achieved without sacrificing learning performance for general convex Lipschitz losses.
JoVA combines two VAEs to learn user and item representations for better recommendation.
problem Collaborative filtering with implicit feedback.
method Joint Variational Autoencoders (JoVA) with a hinge-based pairwise loss function (JoVA-Hinge).
result JoVA-Hinge outperforms state-of-the-art methods in top-k recommendation.
A new procedure for learning cost-sensitive SVM(CS-SVM) classifiers is proposed. The SVM hinge loss is extended to the cost sensitive setting, and the CS-SVM is derived as the minimizer of the associated risk. The extension of the hinge loss draws on recent connections between risk minimization and probability elicitat…
Introduces Soft-SVM for binary classification bridging logistic and SVM.
problem Data separability issues in binary classification.
method Soft-SVM regression using convex relaxation of hinge loss with softness and class-separation parameters.
result Soft-SVM performs well in classification and prediction errors.
Proposes SOVR loss to improve adversarial robustness by increasing logit margins.
problem Adversarial training's difficulty in robustness against sophisticated attacks.
method Introduces SOVR loss function that switches from cross-entropy to one-vs-the-rest loss for important samples.
result SOVR loss increases logit margins of important samples, improving robustness against Auto-Attack.
Study efficient learning of halfspaces with constant noise tolerance.
problem Learning halfspaces in the presence of both instance and label corruption.
method Develops an algorithm to minimize reweighted hinge loss for robustness.
result Achieves constant noise tolerance for halfspace learning.
New IRLS algorithms for SVM fitting via MM approach.
problem Fitting support vector machines (SVMs) via quadratic programming.
method Majorization--Minimization (MM) paradigm for iteratively-reweighted least-squares (IRLS) algorithms.
result IRLS algorithms for SVM risk minimization problems with various losses and penalties.
A new L2D system produces calibrated probabilities of expert correctness without sacrificing accuracy.
problem Calibration of learning to defer systems for safety.
method One-vs-all classifiers with a consistent surrogate loss function.
result Proposes a calibrated L2D system that outperforms existing methods in accuracy and calibration.
New framework enhances neural network robustness against adversarial attacks.
problem Vulnerability of deep neural networks to small perturbations.
method Integrates Lipschitz constraint using optimal transport and hinge regularization.
result Proposes a new loss function that certifies adversarial robustness.
Paper studies convergence rates from surrogate risk minimizers to Bayes optimal classifier.
problem Analyzing the convergence rates of surrogate risk minimizers to the Bayes optimal classifier.
method Introducing consistency intensity to characterize surrogate loss functions and using it to derive convergence rates.
result Empirical surrogate risk minimizers converge faster to the Bayes optimal classifier under certain conditions.
New approach estimates personalized treatment effects using surrogate losses.
problem Estimating personalized treatment effects with binary outcomes and limited data.
method Proposes surrogate loss functions that incorporate both treatment and control data.
result Minimax support vector machine formulation yields tighter bounds.
A new method uses Shapley values to select important features for classification.
problem Feature selection for improving classification models.
method Classification game with Shapley value apportioning of hinge loss.
result Threshold 0 on SVEA value identifies significant features.
Efficient algorithm reduces online multiclass prediction regret to sqrt(T).
problem Online multiclass prediction with bandit feedback.
method Second-order algorithm with regret bound of O(1/η√T).
result Regret bound holds for a range of loss functions.
MELC uses entropy for multithreshold classification, showing consistency similar to SVM.
problem Consistency of multithreshold linear classifiers.
method Employed multithreshold maximum margin model based on information theory.
result Objective function upper bounds misclassified points, similar to hinge loss.
End-to-end training improves segmental models for speech recognition.
problem Training segmental models is challenging and lacks understanding under different settings and losses.
method Investigated a model class based on recent successful approaches, trained end-to-end with various losses.
result Marginal log loss provides the most consistent strong performance without requiring ground-truth alignments.
New HL-MRFs and PSL models for scalable structured data.
problem Balancing rich structure and scalability in machine learning.
method Introduces HL-MRFs and PSL, unifying three approaches.
result HL-MRFs and PSL enable scalable modeling of rich structured data.
New method learns distances and similarities robustly from noisy data.
problem Scalability and robustness in metric learning for large datasets.
method Robust online Distance-Similarity learning with Rescaled hinge loss.
result Significantly outperforms state-of-the-art methods in noisy data.
Paper tackles regret minimization in partial monitoring with finite actions and stochastic outcomes.
problem Minimizing regret in partial monitoring with limited feedback.
method Inspired by DMED, proposes PM-DMED and PM-DMED-Hinge algorithms.
result PM-DMED-Hinge matches the derived logarithmic regret lower bound, proving optimality.
The paper explores transferability of adversarial examples between convex and 01 loss models, finding non-transferability due to different decision boundaries caused by outliers.
problem Transferability of adversarial examples between convex and 01 loss models.
method Empirical study of transferability between linear 01 loss and convex (hinge) loss models, and between neural networks with different activation functions.
result Adversarial examples are non-transferable between convex and 01 loss models due to different decision boundaries caused by outliers.
Gaptron algorithm reduces mistakes in online multiclass classification.
problem Online multiclass classification with limited information.
method Randomized first-order algorithm exploiting the gap between zero-one loss and surrogate losses.
result First linear time algorithm with O ( K T ) O(K\sqrt{T}) O ( K T ) expected regret. The paper analyzes the dynamics of a simple neural network using a mean-field approach.
problem Understanding the training dynamics of neural networks, especially in classification tasks.
method Developed an analytic theory using a mean-field limit for a simple neural network.
result Explicitly solved the dynamics of a linearly separable dataset with a linear hinge loss.
Study on using random subspaces for ERM with various loss functions.
problem Improving learning accuracy with computational savings from random subspaces.
method Random subspaces of a hypothesis space, considering data-dependent subspaces.
result Unified analysis showing computational efficiency can be improved without performance loss.