This paper improves loss functions for deep learning with noisy labels.
problem Training deep neural networks with noisy labels.
method The paper introduces a normalization technique to make any loss function robust to noisy labels and proposes a framework called Active Passive Loss (APL) to combine robust loss functions.
result The proposed APL framework consistently outperforms state-of-the-art methods, especially under high noise rates.
The paper proves deep learning can be robust with certain loss functions.
problem The robustness of deep learning models under flawed data.
method Empirical-risk minimization with unbounded, Lipschitz-continuous loss functions.
result These loss functions provide efficient prediction under minimal data assumptions.
Proposes a new loss function for robust learning.
problem Creating a robust loss function for machine learning.
method Extended pseudo Huber loss with log-exp transform and logistic function.
result Linear convergence algorithm for minimizer finding.
The paper sets lower bounds for adversarial robustness in multiclass classification.
problem Adversarial robustness in multiclass classification with arbitrary loss functions.
method Dual and barycentric reformulations for robust risk minimization.
result Sharp lower bounds for adversarial risks are computed efficiently.
Novel loss functions improve decision tree learning from noisy data.
problem Training decision trees with noisy labels.
method Introducing distribution losses and a new negative exponential loss.
result The negative exponential loss leads to efficient and robust decision tree learning.
Symmetrizes loss functions to improve neural network robustness against noisy labels.
problem Designing robust loss functions for noisy labels in neural networks.
method Symmetrization of multi-class loss functions, focusing on cross-entropy and unhinged loss.
result The multi-class unhinged loss is the unique convex symmetric loss under suitable assumptions.
Paper introduces a new robust loss function for RL.
problem Heuristic selection of threshold parameters in quantile Huber loss.
method Derived from Wasserstein distance, captures noise in quantile values.
result Enhances robustness against outliers and enables parameter adjustment.
Proposes a new Huber loss combining absolute and quadratic properties.
problem Improving robustness in learning models.
method Introduces a generalized Huber loss with a log-exp transform and provides an efficient minimization algorithm.
result Shows that the new loss function can be minimized efficiently.
Robust loss minimization is an important strategy for handling robust learning issue on noisy labels. Current robust loss functions, however, inevitably involve hyperparameter(s) to be tuned, manually or heuristically through cross validation, which makes them fairly hard to be generally applied in practice. Besides, t…
Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regre…
In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and…
We present a generalization of the Cauchy/Lorentzian, Geman-McClure, Welsch/Leclerc, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and L2 loss functions. By introducing robustness as a continuous parameter, our loss function allows algorithms built around robust loss minimization to be generalized, which imp…
New algorithm ATENT improves adversarial robustness in neural networks.
problem Improving neural network robustness against adversarial attacks.
method Proposes a new loss function with entropic regularization for training robust neural networks.
result ATENT achieves competitive robust classification accuracy on benchmark datasets.
Unhinged loss minimization fails to improve classifier accuracy for simple data.
problem Accuracy of classifiers minimizing the unhinged loss.
method Minimizing the unhinged loss function.
result Minimizing the unhinged loss yields classifiers with accuracy no better than random guessing for simple data.
New method prevents neural network breakdown by combining trimmed loss and variation regularization.
problem Outlier contamination in neural network training.
method Integrates transformed trimmed loss and higher-order variation regularization.
result Ensures robustness to outlier contamination with a high functional breakdown point.
Proposes a new loss function for learning with noisy labels.
problem Improving model learnability with noisy labels.
method Uses generalized Jensen-Shannon divergence as a noise-robust loss function.
result Shows state-of-the-art results on noisy data.
Proposes new loss functions for GANs to improve estimation accuracy and robustness.
problem Improving the training of GANs to achieve more accurate and robust models.
method Introduces Hellinger-type loss functions and analyzes their statistical properties.
result Demonstrates improved estimation accuracy and robustness of the proposed loss functions.
In this paper, we propose a novel {\em e-exponentiated} transformation, 0≤e<1, for loss functions. When the transformation is applied to a convex loss function, the transformed loss function become more robust to outliers. Using a novel generalization error bound, we have theoretically shown that the transforme…
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First H-consistency bounds for comp-sum losses and smooth adversarial comp-sum losses. With the recent advancement in the deep learning technologies such as CNNs and GANs, there is significant improvement in the quality of the images reconstructed by deep learning based super-resolution (SR) techniques. In this work, we propose a robust loss function based on the preservation of edges obtained by the Can…
Deep models can fit noisy labels, but robustness and reliability are still issues.
problem Training deep models with noisy labels leads to unreliable uncertainty quantification.
method Analysis of conditional distribution over noisy labels and evaluation of robust loss functions.
result Strictly proper and robust loss functions preserve accuracy but do not guarantee reliability.
Adapting robust statistics to neural networks, researchers found neural networks can be more robust with certain loss functions.
problem The robustness of neural networks in complex learning tasks.
method Adapting the regression breakdown point from robust statistics to neural networks and comparing different configurations and contamination settings.
result Neural networks can benefit from robust loss functions, as demonstrated in extensive simulations.
The paper develops robust risk measures for uncertain loss positions.
problem Risk assessment for loss positions with uncertain distributions.
method Robust optimized certainty equivalents and generalized quantiles are proposed and analyzed.
result Robust expectiles with specific penalization functions are coherent risk measures.
Develops a new framework for robust regression with EGM.
problem Addressing robust regression with heavy-tailed noise or outliers.
method Empirical gain maximization (EGM) to approximate noise density.
result Unified analysis of robust regression approaches.
Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…
Paper analyzes statistical properties of log-cosh loss function.
problem No statistical analysis of log-cosh loss function in literature.
method Presented statistical properties of log-cosh loss function, compared to Cauchy distribution, and examined various statistical procedures.
result Characterized statistical properties of log-cosh loss function, including distribution, likelihood function, and Fisher information.
New robust loss functions improve matrix completion accuracy.
problem Outliers in data corrupting matrix completion accuracy.
method Developed nonconvex M-estimator functions to down-weight outliers.
result Proposed methods outperform competitors in recovery accuracy and runtime.
AugLoss combines data augmentation and robust loss functions for robust DL models.
problem Robustness against noisy labels and feature distribution shifts.
method Unified data augmentation and robust loss functions.
result AugLoss achieves gains over previous methods in various real-world dataset corruptions.
It is known that Boosting can be interpreted as a gradient descent technique to minimize an underlying loss function. Specifically, the underlying loss being minimized by the traditional AdaBoost is the exponential loss, which is proved to be very sensitive to random noise/outliers. Therefore, several Boosting algorith…
This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the outliers in the data. The analysis of the role of a particular non-convex loss for prediction accur…
Deep networks can memorize random labels; symmetric loss improves this.
problem Deep networks can memorize random labels, ignoring standard regularization.
method Empirical studies with MNIST and CIFAR-10 datasets, formal definition of robustness.
result Symmetric loss function improves network's ability to resist memorization.
Paper studies a robust online learning algorithm for regression.
problem Develops a robust online learning algorithm for regression problems.
method Introduces an online learning algorithm with a robust loss function over RKHS.
result The algorithm achieves optimal convergence rates in mean square and RKHS.
Paper tackles robust deep learning from weakly dependent data with unbounded loss and input.
problem Tackles robust deep learning from weakly dependent data with unbounded loss and input.
method Establishes non-asymptotic bounds for expected excess risk under strong mixing and ψ-weak dependence assumptions. result Derives a relationship between bounds and r, and shows convergence rate close to i.i.d. results for r=∞. Investor optimizes investment and consumption under uncertain market conditions with constraints.
problem Investor optimizes investment and consumption in a stochastic environment with model uncertainty and constraints.
method Robust control problem solved using stochastic Hamilton-Jacobi-Bellman-Isaacs equations, backward stochastic differential equations, and bounded mean oscillation martingale theory.
result Investor incurs utility loss when ignoring model uncertainty, and constraints impact optimal strategy and value function.
We introduce a tunable loss function called α-loss, parameterized by α∈(0,∞], which interpolates between the exponential loss (α=1/2), the log-loss (α=1), and the 0-1 loss (α=∞), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $…
Starting from the requirement that risk measures of financial portfolios should be based on their losses, not their gains, we define the notion of loss-based risk measure and study the properties of this class of risk measures. We characterize loss-based risk measures by a representation theorem and give examples of su…
This paper characterizes and designs loss functions for robust classification with abstention.
problem Ensuring robustness against adversarial attacks and knowing when to abstain from prediction.
method Proposes adversarial robust reject option loss and characterizes surrogates for calibration.
result Shifted Double Ramp Loss and Shifted Double Sigmoid Loss satisfy the calibration conditions.
New expressive losses improve adversarial robustness without sacrificing accuracy.
problem Training networks for robustness at the expense of accuracy.
method Formalizing expressivity, using convex combinations of adversarial attacks and IBP bounds.
result Trivial expressive losses yield state-of-the-art results in various settings.
Robust variable selection for high-dimensional data with missing and measurement errors.
problem Missing data and measurement errors confound data distribution.
method Exponential loss function with inverse probability weighting and additive error models.
result The Atan punishment method improves robust variable selection.
In high-dimensional data, many sparse regression methods have been proposed. However, they may not be robust against outliers. Recently, the use of density power weight has been studied for robust parameter estimation and the corresponding divergences have been discussed. One of such divergences is the γ-divergence a…
AWP improves robustness by flattening weight loss landscape.
problem Improving robustness of deep neural networks against adversarial examples.
method Explicitly regularizes the flatness of weight loss landscape through adversarial weight perturbation.
result AWP forms a double-perturbation mechanism in adversarial training, leading to flatter weight loss landscape.
Exponential Lasso improves Lasso's robustness to outliers and heavy-tailed noise.
problem Lasso's sensitivity to outliers and heavy-tailed noise in high-dimensional statistics.
method Integrates an exponential-type loss function into the Lasso framework.
result Achieves strong statistical convergence rates robust to heavy-tailed contamination.
We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determ…
Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory. In this paper regularized pairwise learning (RPL) methods based on kernels will be investigated. One example is regularized minimization of the error entropy loss which has recently attracted qu…
We present the Tamed Cross Entropy (TCE) loss function, a robust derivative of the standard Cross Entropy (CE) loss used in deep learning for classification tasks. However, unlike other robust losses, the TCE loss is designed to exhibit the same training properties than the CE loss in noiseless scenarios. Therefore, th…
RKUM is an R package for robust kernel-based unsupervised methods.
problem Robust analysis under contaminated or noisy data conditions.
method Robust kernel covariance and cross-covariance operators using generalized loss functions.
result RKUM reduces sensitivity to contamination and effectively identifies outliers.
Gradient boosting algorithms construct a regression predictor using a linear combination of ``base learners''. Boosting also offers an approach to obtaining robust non-parametric regression estimators that are scalable to applications with many explanatory variables. The robust boosting algorithm is based on a two-stag…
Distance/Similarity learning is a fundamental problem in machine learning. For example, kNN classifier or clustering methods are based on a distance/similarity measure. Metric learning algorithms enhance the efficiency of these methods by learning an optimal distance function from data. Most metric learning methods nee…