Label smoothing improves model performance even with noisy labels.
problem Mitigating label noise in deep learning models.
method Examined label smoothing as a technique to cope with label noise and compared it to loss-correction methods.
result Label smoothing is competitive with loss-correction techniques under label noise and beneficial for distillation from noisy data.
Label smoothing improves generalization by controlling generalization loss.
problem Lack of mathematical understanding of label smoothing's effectiveness.
method Proposed a theoretical framework to show how label smoothing controls generalization loss in the label noise setting.
result Predicted an optimal label smoothing point that minimizes generalization loss.
Label smoothing improves model calibration and generalization but harms distillation.
problem Understanding the effects of label smoothing on model calibration and distillation.
method Empirical evaluation and visualization of network representations.
result Label smoothing improves model calibration but harms knowledge distillation.
Pairwise Label Smoothing improves deep model generalization by reducing overconfidence.
problem Improving deep model generalization through regularization.
method PLS smooths labels for pairs of samples, learning distribution mass during training.
result PLS significantly outperforms LS and baseline models, reducing up to 30% classification error.
Label-Smoothing improves adversarial robustness of deep learning models.
problem Improving the robustness of deep learning models against adversarial attacks.
method Label-Smoothing, including adversarial, Boltzmann, and second-best variants, applied to various models and datasets.
result Label-Smoothing enhances adversarial robustness across multiple attacks and datasets.
Paper introduces SLS to improve label smoothing regularization.
problem Improve generalization of neural networks by softening labels.
method Structural Label Smoothing (SLS) to mitigate bias in Bayes error rate.
result Empirical validation shows SLS outperforms traditional label smoothing.
Improved OOD detection using label smoothing and k-NN density estimates.
problem Detecting out-of-distribution examples in classification models.
method Label smoothing and k-NN density estimate on intermediate activations.
result Label smoothing improves OOD detection performance, both theoretically and empirically.
Improves confidence calibration in neural networks by smoothing labels based on class similarity.
problem Improving confidence calibration in deep neural networks for safety-critical applications.
method Proposes a novel label smoothing technique where label values are based on similarities with the reference class, using different similarity measurements.
result Consistently outperforms state-of-the-art calibration techniques on various datasets and network architectures.
Enhances deep networks robustness with data mollification and label smoothing.
problem Improving deep neural networks' robustness against corruptions.
method Coupling data mollification (image noising and blurring) with label smoothing.
result Improved robustness and uncertainty quantification on corrupted image benchmarks.
KGNN-LS improves recommender systems using knowledge graphs and label smoothness.
problem Improving recommender systems through better user-item embeddings.
method KGNN-LS combines knowledge graphs, user-specific embeddings, and label smoothness regularization.
result KGNN-LS outperforms state-of-the-art baselines and handles cold-start scenarios.
In this article, we mathematically study several GAN related topics, including Inception score, label smoothing, gradient vanishing and the -log(D(x)) alternative. --- An advanced version is included in arXiv:1703.02000 "Activation Maximization Generative Adversarial Nets". Please refer Section 6 in 1703.02000 for deta…
The study examines label smoothing to improve confidence calibration in fine-tuned LLMs.
problem Improving confidence calibration in fine-tuned large language models (LLMs) after instruction tuning.
method Examine various open-sourced LLMs, label smoothing, and custom kernel design.
result Label smoothing is effective in maintaining confidence calibration but faces challenges in large vocabulary LLMs.
Study analyzes label smoothing in deep learning optimization.
problem Understanding label smoothing's impact on deep learning optimization.
method Analysis of stochastic gradient descent with label smoothing for non-convex problems.
result Label smoothing can speed up convergence by reducing variance.
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
problem Improving model generalization and performance through self-distillation.
method Interpreting self-distillation as MAP estimation and proposing instance-specific label smoothing.
result Self-distillation enhances model performance by increasing teacher diversity and smoothing predictions.
GS-B3SE improves label shift estimation by smoothing priors on a graph.
problem Label shift adaptation when source and target distributions share conditional but not marginal probabilities.
method Graph-Smoothed Bayesian Black-Box Shift Estimator (GS-B3SE) places Laplacian-Gaussian priors on log-priors and confusion-matrix columns tied by a label-similarity graph. result GS-B3SE produces a tractable posterior with HMC or Newton-CG schemes, proving identifiability, contraction, and robustness. Label smoothing improves model robustness against misspecification.
problem Improving model robustness against model misspecification.
method Introducing modified label smoothing (MLSLR) that maintains consistent probability estimation while modifying the loss function.
result MLSLR exhibits higher robustness against model misspecification than conventional label smoothing.
Proposes a data augmentation method to improve multi-label learning performance.
problem Improving multi-label learning by exploiting label correlations and data augmentation.
method Proposes a novel data augmentation approach that performs clustering on real examples and treats cluster centers as virtual examples, promoting local smoothness through a regularization term.
result Extensive experiments show that the proposed method outperforms state-of-the-art multi-label learning approaches.
Method reweights instances and classes to improve robustness in noisy data.
problem Improving deep learning performance in the presence of label noise.
method Formulates constrained optimization problems to assign importance weights to instances and class labels.
result Significant performance gains observed in benchmark datasets with label noise.
Practically, we are often in the dilemma that the labeled data at hand are inadequate to train a reliable classifier, and more seriously, some of these labeled data may be mistakenly labeled due to the various human factors. Therefore, this paper proposes a novel semi-supervised learning paradigm that can handle both l…
New method makes machine learning models robust to label flipping attacks.
problem Machine learning models are vulnerable to label flipping attacks.
method Randomized smoothing over arbitrary functions to build certifiably robust classifiers.
result Linear classifiers are robust to label flipping attacks with deterministic bounds.
The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data…
Simple regularization methods mimic adversarial training's robustness.
problem Expensive adversarial training for robustness.
method Label smoothing and logit squeezing.
result Achieves strong adversarial robustness without adversarial examples.
Label assignment problems with large state spaces are important tasks especially in computer vision. Often the pairwise interaction (or smoothness prior) between labels assigned at adjacent nodes (or pixels) can be described as a function of the label difference. Exact inference in such labeling tasks is still difficul…
Simple methods boost sound event classifier accuracy by 2.5%.
problem Label noise in sound event classification datasets.
method Label smoothing regularization, mixup, and noise-robust loss functions.
result Simple methods can boost accuracy by up to 2.5%.
Proposes sigmoidF1 loss for multilabel classification, improving performance metrics.
problem Lack of smooth, tractable loss functions for multilabel classification.
method Introduces sigmoidF1, a smooth F1 score surrogate loss function.
result sigmoidF1 outperforms other loss functions on various datasets and metrics.
A new method improves graph-based semi-supervised classification by removing noise and mixed signs.
problem Inaccurate soft labels and noise in graph-based semi-supervised classification.
method Triple-matrix-recovery-based robust auto-weighted label propagation framework (ALP-TMR).
result Improved robustness to noise and outliers in label estimation.
This work proposes a novel method for semi-supervised learning from partially labeled massive network-structured datasets, i.e., big data over networks. We model the underlying hypothesis, which relates data points to labels, as a graph signal, defined over some graph (network) structure intrinsic to the dataset. Follo…
Learning with Label Proportions (LLP) is the problem of recovering the underlying true labels given a dataset when the data is presented in the form of bags. This paradigm is particularly suitable in contexts where providing individual labels is expensive and label aggregates are more easily obtained. In the healthcare…
Logit regularization induces logit clustering, affecting classifier performance.
problem Understanding the mechanism of logit regularization in classification.
method Analysis of logit regularization in linear classification, proving logit clustering leads to Fisher's Linear Discriminant alignment.
result Logit regularization can halve critical sample complexity and induce robust generalization.
IDS improves RLHF by smoothing reward data, enhancing model performance.
problem Reward model performance degrades and overoptimization hinders true objective.
method Iterative Data Smoothing (IDS) updates model and data labels during each epoch.
result IDS outperforms traditional methods in RLHF.
Simple method improves deep classifier accuracy under noisy labels.
problem Training deep classifiers with noisy labels.
method Probabilistic approach using temperature parameterized softmax.
result Improves accuracy, log-likelihood and calibration on noisy datasets.
Adaptive label regularization improves neural network performance.
problem Improving deep neural network performance through label regularization.
method Proposes an adaptive label regularization method that updates label representation online.
result Significant improvement in image recognition and text classification tasks.
A new unsupervised contrastive learning framework improves time series representation learning.
problem Lack of labeled data in time series data.
method Proposes an unsupervised contrastive learning framework using a novel contrastive loss and data augmentation.
result Framework outperforms other approaches on univariate and multivariate time series, and benefits transfer learning.
New method calibrates multi-class predictions efficiently without sacrificing accuracy.
problem Efficiently calibrating multi-class predictions without sacrificing accuracy.
method Formulated robust projected smooth calibration and new recalibration algorithms.
result Achieves strong guarantees for binary classification tasks with polynomial complexity.
CCVAE captures label characteristics in VAEs for better representation learning.
problem Capturing rich label characteristics in VAEs without conflating them with label values.
method Developed CCVAE, a novel VAE model that explicitly captures label characteristics in latent space.
result CCVAE allows for effective and general interventions like smooth traversals and diverse conditional generation.
PML-LFC improves PML by estimating label confidence from both feature and label spaces.
problem PML challenges in real-world scenarios where only some labels are relevant.
method PML-LFC estimates label confidence using feature and label space similarities, training a predictor with these values.
result PML-LFC achieves superior performance on synthetic and real-world datasets.
In applications of supervised learning applied to medical image segmentation, the need for large amounts of labeled data typically goes unquestioned. In particular, in the case of brain anatomy segmentation, hundreds or thousands of weakly-labeled volumes are often used as training data. In this paper, we first observe…
Let M be a smooth closed orientable surface, and let F be the space of Morse functions on M such that at least χ(M)+1 critical points of each function of F are labeled by different labels (enumerated). Endow the space F with C∞-topology. We prove the homotopy equivalence $F\sim R\times{\widetilde{\c…
CutMix training technique improves spatial locality in Vision Transformers.
problem Improving spatial locality in Vision Transformers trained from scratch.
method Comparison of Baseline and Modern training protocols on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
result CutMix training component significantly reduces Mean Attention Distance (MAD) in early layers of Vision Transformers.
Proposes a method for anomaly detection with inexact labels.
problem Handling anomaly detection with inexact labels.
method Trains an anomaly score function using a neural network-based unsupervised method, maximizing the inexact AUC.
result Improves anomaly detection performance with inexact labels and outperforms existing methods.
SGD handles label noise with bounds improving over SGLD.
problem Label noise in non-convex optimization.
method Stochastic gradient descent with uniform dissipativity and smoothness conditions, using Wasserstein distance and algorithmic stability.
result Generalization error bounds with a rate of n−2/3, better than SGLD's n−1/2. Noisy labels often occur in vision datasets, especially when they are obtained from crowdsourcing or Web scraping. We propose a new regularization method, which enables learning robust classifiers in presence of noisy data. To achieve this goal, we propose a new adversarial regularization scheme based on the Wasserstei…
We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adver…
AUC-spec optimizes graph-based SSL for complex label distributions.
problem Training accurate models with scarce labeled data and abundant unlabeled data.
method Computes a low-dimensional representation that maximizes class separation via AUC optimization.
result AUC-spec achieves competitive results on synthetic and real-world datasets.
Unified surrogate loss framework for multi-label learning with strong consistency guarantees.
problem Improving consistency and accounting for label correlations in multi-label learning.
method Introducing multi-label logistic loss and extending it to comprehensive multi-label comp-sum losses, proving strong consistency guarantees for any multi-label loss.
result Unified surrogate loss framework benefiting from strong consistency guarantees for any multi-label loss.
We describe an adversarial learning approach to constrain convolutional neural network training for image registration, replacing heuristic smoothness measures of displacement fields often used in these tasks. Using minimally-invasive prostate cancer intervention as an example application, we demonstrate the feasibilit…
We apply the network Lasso to classify partially labeled data points which are characterized by high-dimensional feature vectors. In order to learn an accurate classifier from limited amounts of labeled data, we borrow statistical strength, via an intrinsic network structure, across the dataset. The resulting logistic …
In this paper we study the existence of solutions for a class of non-linear differential equation on compact Riemannian manifolds. We establish a lower and upper solutions' method to show the existence of a smooth positive solution for the equation (EQ1) \begin{equation} \label{E4} Δu \ + \ a(x)u \ = \ f(x)F(u) \ + \ h…