Paper establishes comparison theorems for large-margin learning.
problem Data piling issue in high-dimension and low-sample size SVM.
method Large-margin unified machines (LUM) loss functions.
result New comparison theorems for all LUM loss functions.
Introduces Fitzpatrick losses, tighter than Fenchel-Young losses.
problem Improving loss functions for machine learning.
method Introduces Fitzpatrick losses based on the Fitzpatrick function.
result Fitzpatrick losses are tighter than Fenchel-Young losses.
Classification is the most important process in data analysis. However, due to the inherent non-convex and non-smooth structure of the zero-one loss function of the classification model, various convex surrogate loss functions such as hinge loss, squared hinge loss, logistic loss, and exponential loss are introduced. T…
This paper improves loss functions for deep learning with noisy labels.
problem Training deep neural networks with noisy labels.
method The paper introduces a normalization technique to make any loss function robust to noisy labels and proposes a framework called Active Passive Loss (APL) to combine robust loss functions.
result The proposed APL framework consistently outperforms state-of-the-art methods, especially under high noise rates.
Paper explores connections between loss functions and consistency in binary classification and regression.
problem Consistency in binary classification and regression applications.
method Characterization of conformable loss functions and derivation of a new Huber-type loss function.
result Margin-based loss functions are equivalent to loss functions of squared standardized logistic regression residuals.
Proposes a generalized XGBoost method for nonconvex loss functions.
problem Limited to convex loss functions in XGBoost.
method Extends XGBoost to use nonconvex loss functions and multivariate loss functions.
result Generalized XGBoost method can model multiple parameters in various distributions.
The study explores loss functions for learning distributions, finding the log loss and others are sufficient under certain conditions.
problem Understanding loss functions for distribution learning and density estimation.
method An axiomatic approach to design loss functions, proposing criteria and showing that no single loss function satisfies all criteria.
result No loss function satisfies all criteria, but the log loss and others do under the condition of candidate distributions being calibrated.
Paper connects loss functions and t-norms for faster deep learning convergence.
problem Improving convergence rates in deep learning models.
method Interprets loss functions through t-norms and generator functions.
result Derives a general relation between loss functions and t-norms leading to faster convergence.
New loss functions based on f-divergences improve language model performance.
problem Improving multiclass classification and language modeling performance.
method Constructing new convex loss functions using f-divergences and deriving an operator for computation.
result The α-divergence loss function with α=1.5 performs well across various tasks. The paper proves deep learning can be robust with certain loss functions.
problem The robustness of deep learning models under flawed data.
method Empirical-risk minimization with unbounded, Lipschitz-continuous loss functions.
result These loss functions provide efficient prediction under minimal data assumptions.
SoftAdapt dynamically adjusts loss weights for multi-part functions.
problem Slow convergence and poor weight selection for multi-part loss functions.
method SoftAdapt dynamically changes weights based on live performance statistics.
result Improved convergence and better weight selection for multi-part loss functions.
New loss function optimization improves training speed and accuracy.
problem Optimizing neural network performance through loss functions.
method Genetic Loss-function Optimization (GLO) using genetic programming and CMA-ES.
result GLO loss functions lead to better performance with fewer training steps.
In this paper, we propose a novel {\em e-exponentiated} transformation, 0≤e<1, for loss functions. When the transformation is applied to a convex loss function, the transformed loss function become more robust to outliers. Using a novel generalization error bound, we have theoretically shown that the transforme…
Multi-output is essential in machine learning that it might suffer from nonconforming residual distributions, i.e., the multi-output residual distributions are not conforming to the expected distribution. In this paper, we propose "Wrapped Loss Function" to wrap the original loss function to alleviate the problem. This…
Proposes a new loss function for robust learning.
problem Creating a robust loss function for machine learning.
method Extended pseudo Huber loss with log-exp transform and logistic function.
result Linear convergence algorithm for minimizer finding.
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First H-consistency bounds for comp-sum losses and smooth adversarial comp-sum losses. Novel loss functions improve decision tree learning from noisy data.
problem Training decision trees with noisy labels.
method Introducing distribution losses and a new negative exponential loss.
result The negative exponential loss leads to efficient and robust decision tree learning.
Study compares metric learning loss functions for speaker verification.
problem Comparing metric learning loss functions for end-to-end speaker verification.
method Cross entropy loss, cosine loss, angular margin loss, center loss, contrastive loss, triplet loss.
result Additive angular margin loss outperforms other loss functions.
We present α-loss, α∈[1,∞], a tunable loss function for binary classification that bridges log-loss (α=1) and 0-1 loss (α=∞). We prove that α-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal y…
Efficiently learns loss functions for better model performance.
problem Finding an optimal loss function for model validation.
method Asymptotically optimal anytime algorithm in worst case, provably efficient in idealized case.
result Significantly faster than state-of-the-art alternatives for tuning loss function hyperparameters.
Paper introduces new loss functions for Siamese networks using FDA.
problem Training Siamese networks with improved loss functions.
method Proposes Fisher Discriminant Triplet (FDT) and Fisher Discriminant Contrastive (FDC) loss functions based on FDA.
result Shows effectiveness of FDT and FDC on MNIST and histopathology datasets.
New loss function equivalence reveals PER's uniform sampling can be improved.
problem Improving Prioritized Experience Replay (PER) for better learning efficiency.
method Transforming non-uniformly sampled data loss functions into uniformly sampled ones.
result Some environments can replace PER with a new loss function without performance loss.
Symmetrizes loss functions to improve neural network robustness against noisy labels.
problem Designing robust loss functions for noisy labels in neural networks.
method Symmetrization of multi-class loss functions, focusing on cross-entropy and unhinged loss.
result The multi-class unhinged loss is the unique convex symmetric loss under suitable assumptions.
The paper explores how different loss functions impact reinforcement learning algorithms.
problem Improving reinforcement learning algorithms by optimizing loss functions.
method Comprehensive survey on loss functions in reinforcement learning, proving the benefits of specific loss functions.
result Binary cross-entropy loss leads to first-order bounds and is more efficient than squared loss.
Study proposes a differentiable surrogate loss function for optimizing Fβ score in binary classification with imbalanced data.
problem Non-differentiability of Fβ score makes it unsuitable for optimization by gradient-based learning. method Investigated relationship between Fβ score and loss functions, proposed a differentiable surrogate loss function. result Gradient paths of the proposed surrogate Fβ loss function approximate the gradient paths of the Fβ score. Improved CNN for HCCR with new loss function and ranking method.
problem Loss of inter-class information in traditional CNN models for HCCR.
method Combining cross entropy with a new similarity ranking function (Average variance similarity) as loss function.
result New loss function (SoftMax cross entropy with Average variance similarity) achieves highest accuracy in HCCR.
Paper introduces a new robust loss function for RL.
problem Heuristic selection of threshold parameters in quantile Huber loss.
method Derived from Wasserstein distance, captures noise in quantile values.
result Enhances robustness against outliers and enables parameter adjustment.
Develops a new theory of loss functions for statistical machine learning.
problem Evaluation of solutions in binary and multiclass classification problems.
method Defines loss functions as subgradients of support functions of convex sets, enabling a calculus of losses.
result Provides a novel perspective on losses and develops a calculus that interpolates between different losses.
The paper extends mixability theory to function-valued forecasts, proving various loss functions are mixable.
problem Efficient aggregation of functional and probabilistic forecasts in online prediction games.
method Adapting mixable and exponentially concave loss functions to function-valued forecasts.
result Various loss functions used for probabilistic forecasting are mixable (exp-concave).
Paper proposes a method to minimize non-differentiable loss functions.
problem Minimizing non-differentiable and non-decomposable loss functions.
method Learn smooth relaxations of true losses through surrogate neural networks, then optimize jointly with the prediction model.
result Empirical results show the efficiency of learning surrogate losses.
Tilting loss functions improves machine learning performance.
problem Improving machine learning models, especially in under- and over-parameterized networks.
method Using evolving loss functions that emphasize different classes cyclically.
result Dynamical loss functions lead to better generalization and stability in training.
Regularizers change the geometric properties of loss functions in neural networks.
problem Understanding how different regularizers affect the geometric properties of loss functions in neural networks.
method Examined several regularizers, including weight decay, to determine if the regularized loss function becomes Morse.
result For certain regularizers, the regularized loss function becomes Morse, indicating a change in geometric properties.
This paper explores ratio-based loss functions for machine learning.
problem Margin-based and distance-based loss functions for classification and regression.
method Investigation of ratio-based loss functions' properties.
result Proposed new ratio-based loss functions for regression.
VIABLE learns a loss function for better few-shot learning.
problem Few-shot learning underfits with standard loss functions.
method Meta-learning to learn a differentiable loss function.
result Learning a relational loss function improves performance and sample efficiency.
In this paper we refine the process of computing calibration functions for a number of multiclass classification surrogate losses. Calibration functions are a powerful tool for easily converting bounds for the surrogate risk (which can be computed through well-known methods) into bounds for the true risk, the probabili…
Generative adversarial nets (GANs) are widely used to learn the data sampling process and their performance may heavily depend on the loss functions, given a limited computational budget. This study revisits MMD-GAN that uses the maximum mean discrepancy (MMD) as the loss function for GAN and makes two contributions. F…
Adma proposes a flexible loss function for neural networks.
problem Static loss functions limit neural network performance.
method Introduces a flexible loss function that adapts to ANN complexity and data distribution.
result Flexible loss function achieves state-of-the-art performance.
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce mi…
Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regre…
Paper analyzes statistical properties of log-cosh loss function.
problem No statistical analysis of log-cosh loss function in literature.
method Presented statistical properties of log-cosh loss function, compared to Cauchy distribution, and examined various statistical procedures.
result Characterized statistical properties of log-cosh loss function, including distribution, likelihood function, and Fisher information.
With the recent advancement in the deep learning technologies such as CNNs and GANs, there is significant improvement in the quality of the images reconstructed by deep learning based super-resolution (SR) techniques. In this work, we propose a robust loss function based on the preservation of edges obtained by the Can…
New method finds better loss functions for neural nets.
problem Finding effective loss functions for deep neural networks.
method Optimizes multivariate Taylor polynomial parameterizations using CMA-ES.
result TaylorGLO finds loss functions that outperform existing methods.
Paper proposes new loss functions for training energy networks.
problem Challenges in computing gradients for training energy networks.
method Proposes generalized Fenchel-Young losses for efficient gradient computation.
result Demonstrates the calibration of excess risk for linear-concave energies.
In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and…
This work classifies SOC loss functions based on their gradient properties.
problem Optimizing noisy systems in stochastic optimal control.
method Grouping loss functions into classes with the same gradient expectation.
result Different loss functions have the same optimization landscape but differ in gradient variance.
Deep networks can memorize random labels; symmetric loss improves this.
problem Deep networks can memorize random labels, ignoring standard regularization.
method Empirical studies with MNIST and CIFAR-10 datasets, formal definition of robustness.
result Symmetric loss function improves network's ability to resist memorization.
New α-divergence loss function improves neural density ratio estimation.
problem Optimization challenges in existing DRE methods, especially overfitting and high sample requirements.
method Derived α-divergence loss function (α-Div) for neural density ratio estimation. result The α-divergence loss function (α-Div) offers stable and effective optimization for DRE. Proposes a new loss function for learning with noisy labels.
problem Improving model learnability with noisy labels.
method Uses generalized Jensen-Shannon divergence as a noise-robust loss function.
result Shows state-of-the-art results on noisy data.