Quantum machine learning uses quantum cross entropy to minimize loss, but measurement loss affects this process.
problem Quantum machine learning's loss minimization through cross entropy is affected by measurement outcomes.
method Defined quantum cross entropy, proved its lower bounds, and investigated its relation to quantum fidelity and likelihood.
result Quantum cross entropy is lower-bounded by negative log-likelihood when derived from quantum data, but measurement outcomes can cause loss.
Cross-entropy loss linked to metric learning, outperforming complex pairwise losses.
problem Improving metric learning performance without complex optimization schemes.
method Theoretical analysis linking cross-entropy to pairwise losses, showing cross-entropy as an upper bound and equivalent to mutual information maximization.
result Minimizing cross-entropy is equivalent to maximizing mutual information, leading to state-of-the-art performance.
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
problem Lack of theoretical relationship between softmax cross-entropy and negative sampling loss functions in knowledge graph embedding.
method Used Bregman divergence to provide a unified interpretation of the two loss functions.
result Theoretical findings for fair comparison of softmax cross-entropy and negative sampling are derived.
PAC-Bayesian bounds for MLPs with cross entropy loss validated.
problem Generalization bounds for MLPs with cross entropy loss.
method Introduced probabilistic explanations and proved PAC-Bayesian bounds using ELBO.
result MLPs with cross entropy loss inherently guarantee PAC-Bayesian generalization bounds.
New method detects changes by maximizing cross-entropy, outperforming existing techniques.
problem Detecting abrupt changes in data streams without labeled examples.
method Maximizes cross-entropy between segments to find change points, using dynamic programming.
result Outperforms three state-of-the-art approaches on challenging datasets.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Square loss performs comparably or better than cross-entropy in neural architectures for various tasks.
problem The superiority of cross-entropy loss over square loss in classification tasks is debated.
method Comparison of several neural architectures on NLP, ASR, and computer vision datasets using both loss functions.
result Square loss often produces better results in the majority of tasks, especially in NLP and ASR.
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First H-consistency bounds for comp-sum losses and smooth adversarial comp-sum losses. Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
Study Transformer layers under cross-entropy training using mean field control.
problem Understanding the behavior of Transformer layers in cross-entropy training.
method Continuous-depth mean field control analysis, treating depth as time and layer parameters as controls.
result Derivation of a Pontryagin condition for the limiting population problem, involving the softmax residual.
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from co…
Proposes a method to solve deep neural networks' local minimum problem.
problem Local minimum problem in deep neural networks training.
method Transforms cross-entropy loss into risk-averse error criterion, adjusts RSI, and uses convexity region.
result Trained deep learning machine is expected to be inside a global minimum's attraction basin.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
Convolution Neural Networks (CNN) have recently achieved state-of-the art performance on handwritten Chinese character recognition (HCCR). However, most of CNN models employ the SoftMax activation function and minimize cross entropy loss, which may cause loss of inter-class information. To cope with this problem, we pr…
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
Improved CEM for fast real-time planning in high-dimensional control tasks.
problem Sampling inefficiency of CEM in real-time planning.
method Novel additions to CEM including temporally-correlated actions and memory.
result 2.7-22x less samples and 1.2-10x performance increase.
We present the Tamed Cross Entropy (TCE) loss function, a robust derivative of the standard Cross Entropy (CE) loss used in deep learning for classification tasks. However, unlike other robust losses, the TCE loss is designed to exhibit the same training properties than the CE loss in noiseless scenarios. Therefore, th…
Generative Cross-Entropy improves classification with fewer labels.
problem Limited sample efficiency of cross-entropy loss in data-scarce scenarios.
method Proposes Generative Cross-Entropy (GenCE), a new loss function that incorporates generative principles into a standard discriminative network.
result Generative Cross-Entropy outperforms traditional cross-entropy loss across various datasets and conditions.
Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss fun…
This paper shows using classification instead of regression improves deep RL scalability.
problem Challenges in training value functions for large networks in deep RL.
method Used categorical cross-entropy loss instead of mean squared error regression.
result Significant improvements in performance and scalability across various domains.
New ACE cost function encourages diversity in neural networks.
problem Training multiple classifiers with controlled diversity.
method Mathematical derivation and gradient control.
result ACE yields better ensemble results than vanilla.
Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy lab…
SimLoss improves classification by considering class similarities.
problem Equal punishment of all misclassifications in CCE.
method Integrates class similarities into Categorical Cross Entropy using matrices constructed from task-specific knowledge.
result Significant improvements over CCE on Age Estimation and Image Classification.
Develops a mean-field theory for multi-head self-attention under cross-entropy training.
problem Mean-field analysis of multi-head self-attention under cross-entropy training.
method Mean-field theory for a simplified single-layer causal multi-head self-attention model.
result Proves a static finite-head approximation bound for the optimal risk.
We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…
This work shows that supervised contrastive learning achieves similar results to cross-entropy but requires more iterations.
problem The question of whether there are fundamental differences in representation geometry between supervised contrastive learning and cross-entropy.
method The authors prove that both losses attain their minimum when representations of each class collapse to the vertices of a regular simplex, and they empirically validate this finding.
result Supervised contrastive learning requires more iterations to reach a close-to-optimal state compared to cross-entropy, indicating different optimization behavior.
Deep learning has been shown to achieve impressive results in several domains like computer vision and natural language processing. A key element of this success has been the development of new loss functions, like the popular cross-entropy loss, which has been shown to provide faster convergence and to reduce the vani…
In this paper, we focus on the separability of classes with the cross-entropy loss function for classification problems by theoretically analyzing the intra-class distance and inter-class distance (i.e. the distance between any two points belonging to the same class and different classes, respectively) in the feature s…
The R Package CEC performs clustering based on the cross-entropy clustering (CEC) method, which was recently developed with the use of information theory. The main advantage of CEC is that it combines the speed and simplicity of k-means with the ability to use various Gaussian mixture models and reduce unnecessary cl…
New method controls classifier guidance in diffusion models.
problem Improving classifier guidance in diffusion models.
method Cross-entropy control of classifier gradients.
result Effective guidance vectors with mean squared error O(dε). New analysis shows how cross-entropy training shapes attention in transformers.
problem Understanding how gradient-based learning creates the required internal geometry in transformers.
method Developed a first-order analysis of cross-entropy training effects on attention scores and values in a transformer attention head.
result Introduced an advantage-based routing law and responsibility-weighted update for attention scores and values, respectively.
Researchers propose better probabilistic models for deep learning.
problem Using cross-entropy loss for non-categorical data.
method Introducing continuous-categorical distribution and proposing probabilistic alternatives.
result Potential for outperformance in deep learning models with proper probabilistic treatment.
Proposes SOVR loss to improve adversarial robustness by increasing logit margins.
problem Adversarial training's difficulty in robustness against sophisticated attacks.
method Introduces SOVR loss function that switches from cross-entropy to one-vs-the-rest loss for important samples.
result SOVR loss increases logit margins of important samples, improving robustness against Auto-Attack.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
This paper extends neural collapse to imbalanced data under cross-entropy loss.
problem Analyzing neural collapse in deep networks with imbalanced data.
method Using the unconstrained feature model and cross-entropy loss, the paper studies neural collapse in imbalanced datasets.
result Feature vectors within the same class collapse to a single mean vector, but angles between them depend on sample size.
ANTIDOTE reduces noisy labels influence during learning.
problem Learning with noisy labels.
method Information-divergence neighborhood relaxation and adversarial training.
result ANTIDOTE outperforms standard cross-entropy loss in noisy label settings.
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing…
We propose a permutation-invariant loss function designed for the neural networks reconstructing a set of elements without considering the order within its vector representation. Unlike popular approaches for encoding and decoding a set, our work does not rely on a carefully engineered network topology nor by any addit…
Mutual information is widely applied to learn latent representations of observations, whilst its implication in classification neural networks remain to be better explained. We show that optimising the parameters of classification neural networks with softmax cross-entropy is equivalent to maximising the mutual informa…
Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise or tripletwise similarity constraints while others take advantage of structured similarity information among multiple data points. In this work, we approach deep metric l…
We extend recent work (Brehmer, et. al., 2018) that use neural networks as surrogate models for likelihood-free inference. As in the previous work, we exploit the fact that the joint likelihood ratio and joint score, conditioned on both observed and latent variables, can often be extracted from an implicit generative m…
This paper shows cross-entropy can recover latent structures in supervised learning.
problem Understanding why supervised learning works well and how models learn interpretable factors of variation.
method Extending identifiability results to parametric instance discrimination, proving cross-entropy minimization can recover latent structures up to linear transformations.
result Models trained with cross-entropy can learn representations of ground-truth factors of variation up to a linear transformation.
We empirically investigate the (negative) expected accuracy as an alternative loss function to cross entropy (negative log likelihood) for classification tasks. Coupled with softmax activation, it has small derivatives over most of its domain, and is therefore hard to optimize. A modified, leaky version is evaluated on…
This paper reformulates Fβ for better model performance and interpretation.
problem Optimizing model performance and interpretation using Fβ metric. method Reformulate Fβ metric to facilitate statistical distributions and dynamic penalty weights. result Better and interpretable results with a 14% boost in F1 score for IMDB data. Study of focal-entropy for class-imbalanced classification.
problem Understanding the focal-loss in class-imbalanced settings.
method Distributional viewpoint and information-theoretic analysis of focal-entropy.
result Focal-entropy minimizer exists and departs from data distribution.
Paper improves policy updates in reinforcement learning to speed up learning.
problem Slow learning and unlearning in policy optimization.
method Introduces a novel gradient update and a modified policy update.
result Proves modified policy update converges to global optimality.