Designs a new generative model for images using CNNs and improves generalization and performance.
problem Improving image generation and generalization in CNNs.
method Deconvolutional Generative Model (DGM) using a conjugate prior and Rendering Path Normalization (RPN).
result Improves generalization and performance in semi-supervised and supervised learning tasks.
New algorithm identifies optimal subtrees in fixed-budget tree search.
problem Identifying optimal subtrees in fixed-budget Monte Carlo Tree Search.
method ε-agnostic algorithm for max-min action identification.
result Misidentification probability decays exponentially with sample size.
Quantum machine learning uses quantum cross entropy to minimize loss, but measurement loss affects this process.
problem Quantum machine learning's loss minimization through cross entropy is affected by measurement outcomes.
method Defined quantum cross entropy, proved its lower bounds, and investigated its relation to quantum fidelity and likelihood.
result Quantum cross entropy is lower-bounded by negative log-likelihood when derived from quantum data, but measurement outcomes can cause loss.
A new method solves the projection robust Wasserstein distance problem efficiently.
problem Computing the projection robust Wasserstein distance is challenging due to the curse of dimensionality.
method Riemannian block coordinate descent (RBCD) method to solve the regularized max-min problem over the Stiefel manifold.
result RBCD method significantly improves the complexity of obtaining an ε-stationary point compared to existing methods.
Max-min margin Markov networks improve consistency in structured prediction.
problem Statistical inconsistency in max-margin methods for structured prediction.
method Defining a max-min margin formulation to overcome statistical inconsistency.
result Proves consistency and provides an explicit algorithm with finite sample generalization bounds.
New bandit problem for finding best group of arms with worst mean reward.
problem Finding the best group of arms with the worst mean reward in overlapping groups.
method Two algorithms based on successive elimination and robust optimization.
result Upper bounds on the number of samples to find max-min optimal or near-optimal group.
Let M be a stratum of a compact stratified space A. It is equipped with a general adapted metric g, which is slightly more general than the adapted metrics of Nagase and Brasselet-Hector-Saralegi. In particular, g has a general type, which is an extension of the type of an adapted metric. A restriction on this …
Cross-entropy loss linked to metric learning, outperforming complex pairwise losses.
problem Improving metric learning performance without complex optimization schemes.
method Theoretical analysis linking cross-entropy to pairwise losses, showing cross-entropy as an upper bound and equivalent to mutual information maximization.
result Minimizing cross-entropy is equivalent to maximizing mutual information, leading to state-of-the-art performance.
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
problem Lack of theoretical relationship between softmax cross-entropy and negative sampling loss functions in knowledge graph embedding.
method Used Bregman divergence to provide a unified interpretation of the two loss functions.
result Theoretical findings for fair comparison of softmax cross-entropy and negative sampling are derived.
PAC-Bayesian bounds for MLPs with cross entropy loss validated.
problem Generalization bounds for MLPs with cross entropy loss.
method Introduced probabilistic explanations and proved PAC-Bayesian bounds using ELBO.
result MLPs with cross entropy loss inherently guarantee PAC-Bayesian generalization bounds.
Tamed Cross Entropy (TCE) loss outperforms standard CE loss in noisy classification tasks.
problem Improving classification performance in noisy data scenarios.
method Introducing Tamed Cross Entropy (TCE) loss, a derivative of Cross Entropy (CE) loss.
result TCE loss outperforms CE loss in all tested noisy classification scenarios.
Study Transformer layers under cross-entropy training using mean field control.
problem Understanding the behavior of Transformer layers in cross-entropy training.
method Continuous-depth mean field control analysis, treating depth as time and layer parameters as controls.
result Derivation of a Pontryagin condition for the limiting population problem, involving the softmax residual.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
New method detects changes by maximizing cross-entropy, outperforming existing techniques.
problem Detecting abrupt changes in data streams without labeled examples.
method Maximizes cross-entropy between segments to find change points, using dynamic programming.
result Outperforms three state-of-the-art approaches on challenging datasets.
Square loss performs comparably or better than cross-entropy in neural architectures for various tasks.
problem The superiority of cross-entropy loss over square loss in classification tasks is debated.
method Comparison of several neural architectures on NLP, ASR, and computer vision datasets using both loss functions.
result Square loss often produces better results in the majority of tasks, especially in NLP and ASR.
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First H-consistency bounds for comp-sum losses and smooth adversarial comp-sum losses. Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
Improved CNN for HCCR with new loss function and ranking method.
problem Loss of inter-class information in traditional CNN models for HCCR.
method Combining cross entropy with a new similarity ranking function (Average variance similarity) as loss function.
result New loss function (SoftMax cross entropy with Average variance similarity) achieves highest accuracy in HCCR.
Paper solves a max-min game for complex performance benchmarks.
problem Max-min portfolio game with complex performance benchmarks.
method Solves a max-min game for complex performance benchmarks using the axiom of choice.
result Exact maximin strategy found for arbitrary performance benchmarks.
Entropy measures geodesic flow complexity.
problem Measuring complexity of geodesic flows on manifolds.
method Introduced barcode entropy to measure exponential growth rate of not-too-short bars in Morse-theoretic barcodes.
result Barcode entropy bounds topological entropy and vice versa.
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from co…
Paper proposes SL to improve DNN learning with noisy labels.
problem Learning with noisy labels in deep neural networks.
method Symmetric Cross Entropy (SL) with Reverse Cross Entropy (RCE).
result SL outperforms state-of-the-art methods on various datasets.
Proposes a method to solve deep neural networks' local minimum problem.
problem Local minimum problem in deep neural networks training.
method Transforms cross-entropy loss into risk-averse error criterion, adjusts RSI, and uses convexity region.
result Trained deep learning machine is expected to be inside a global minimum's attraction basin.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
Softmax cross-entropy optimizes mutual information in neural networks.
problem Understanding the relationship between mutual information and classification neural networks.
method Demonstrated that optimizing softmax cross-entropy maximizes mutual information between inputs and labels.
result Softmax cross-entropy can approximate mutual information and highlight relevant image regions.
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing…
Study of focal-entropy for class-imbalanced classification.
problem Understanding the focal-loss in class-imbalanced settings.
method Distributional viewpoint and information-theoretic analysis of focal-entropy.
result Focal-entropy minimizer exists and departs from data distribution.
Structured entropy improves classification performance on structured targets.
problem Cross-entropy loss fails to account for target variable structure.
method Proposes structured entropy, a generalization of entropy using random partitions.
result Structured cross-entropy loss yields better results on classification problems with known structure.
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
Study on bit threads and their locking properties in holographic spacetimes.
problem Understanding the conditions under which regions can be locked in holographic spacetimes.
method Investigation of different density bounds and their implications on the locking of regions.
result Non-crossing regions can be locked under the most stringent bound, but crossing regions cannot.
Improved CEM for fast real-time planning in high-dimensional control tasks.
problem Sampling inefficiency of CEM in real-time planning.
method Novel additions to CEM including temporally-correlated actions and memory.
result 2.7-22x less samples and 1.2-10x performance increase.
Generative Cross-Entropy improves classification with fewer labels.
problem Limited sample efficiency of cross-entropy loss in data-scarce scenarios.
method Proposes Generative Cross-Entropy (GenCE), a new loss function that incorporates generative principles into a standard discriminative network.
result Generative Cross-Entropy outperforms traditional cross-entropy loss across various datasets and conditions.
Proposes a cross entropy loss for better ranking algorithms.
problem Improving the theoretical understanding and performance of ranking algorithms.
method Introduces a cross entropy-based loss function that is a convex bound on NDCG and consistent with NDCG.
result Empirically, the proposed method outperforms existing algorithms in quality and robustness.
The paper analyzes how well classes are separated in neural network feature space.
problem Understanding class separability in neural network feature space.
method Theoretical analysis of intra-class and inter-class distances in feature space.
result A lower bound for the probability of inter-class distance being greater than intra-class distance as a function of loss value.
This paper shows using classification instead of regression improves deep RL scalability.
problem Challenges in training value functions for large networks in deep RL.
method Used categorical cross-entropy loss instead of mean squared error regression.
result Significant improvements in performance and scalability across various domains.
New ACE cost function encourages diversity in neural networks.
problem Training multiple classifiers with controlled diversity.
method Mathematical derivation and gradient control.
result ACE yields better ensemble results than vanilla.
ICE proposes a new loss function for deep metric learning.
problem Deep metric learning from instance-level matching distribution.
method Instance Cross Entropy (ICE) loss function.
result ICE outperforms existing methods on real-world benchmarks.
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using …
SimLoss improves classification by considering class similarities.
problem Equal punishment of all misclassifications in CCE.
method Integrates class similarities into Categorical Cross Entropy using matrices constructed from task-specific knowledge.
result Significant improvements over CCE on Age Estimation and Image Classification.
Develops a mean-field theory for multi-head self-attention under cross-entropy training.
problem Mean-field analysis of multi-head self-attention under cross-entropy training.
method Mean-field theory for a simplified single-layer causal multi-head self-attention model.
result Proves a static finite-head approximation bound for the optimal risk.
We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…
Cosine loss improves CNN performance on small datasets.
problem Training CNNs from scratch on small datasets without pre-training.
method Used cosine loss instead of cross-entropy loss.
result Accuracy on CUB-200-2011 dataset is 30% higher with cosine loss.
A new policy improvement method using CEM for Actor-Critic.
problem Improving policy efficiency and robustness in reinforcement learning.
method Greedy Actor-Critic (Greedy AC) using Conditional Cross-Entropy Method (CCEM).
result Greedy AC outperforms Soft Actor-Critic and is less sensitive to entropy regularization.
This work shows that supervised contrastive learning achieves similar results to cross-entropy but requires more iterations.
problem The question of whether there are fundamental differences in representation geometry between supervised contrastive learning and cross-entropy.
method The authors prove that both losses attain their minimum when representations of each class collapse to the vertices of a regular simplex, and they empirically validate this finding.
result Supervised contrastive learning requires more iterations to reach a close-to-optimal state compared to cross-entropy, indicating different optimization behavior.
REMEDI improves neural entropy estimation across various tasks.
problem Challenges in estimating information theoretic quantities in high-dimensional data.
method Combines minimization of cross-entropy with estimation of deviation from data density.
result Improves accuracy in entropy estimation on synthetic and natural data.
The R Package CEC performs clustering based on the cross-entropy clustering (CEC) method, which was recently developed with the use of information theory. The main advantage of CEC is that it combines the speed and simplicity of k-means with the ability to use various Gaussian mixture models and reduce unnecessary cl…