This paper reformulates Fβ for better model performance and interpretation.
problem Optimizing model performance and interpretation using Fβ metric. method Reformulate Fβ metric to facilitate statistical distributions and dynamic penalty weights. result Better and interpretable results with a 14% boost in F1 score for IMDB data. Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
This work extends implicit bias analysis to multiclass classification using a new loss framework.
problem The implicit bias of gradient descent on multiclass data without explicit regularization.
method Employing the PERM framework to introduce a multiclass extension of the exponential tail property.
result Extended implicit bias result to multiclass classification using a new loss framework.
Cubic predicts stock market indices by fusing stock latent embeddings and converting to binary classification.
problem Challenges in predicting stock market indices due to isolated time series treatment and simple regression.
method Fusion of stock latent embeddings, binary encoding classification, and confidence-guided prediction.
result Cubic outperforms state-of-the-art baselines in stock index prediction tasks.
The paper explores how different loss functions impact reinforcement learning algorithms.
problem Improving reinforcement learning algorithms by optimizing loss functions.
method Comprehensive survey on loss functions in reinforcement learning, proving the benefits of specific loss functions.
result Binary cross-entropy loss leads to first-order bounds and is more efficient than squared loss.
Optimizes binary regression models with gradient ascent-descent methods.
problem Regression problems with binary weights in quantized learning and digital communication.
method Maximin optimization using gradient ascent-descent methods.
result The approach is optimal in linear regression with low noise and robust regression with few outliers.
Paper proposes mutual information learning for deep learning classifiers.
problem Overfitting in deep learning models.
method Mutual information learning framework to train classifiers.
result MILCs achieve better generalization than conditional entropy classifiers.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
This paper considers fair probabilistic binary classification where the outputs of primary interest are predicted probabilities, commonly referred to as scores. We formulate the problem of transforming scores to satisfy fairness constraints that are linear in conditional means of scores while minimizing a cross-entropy…
Quantum machine learning uses quantum cross entropy to minimize loss, but measurement loss affects this process.
problem Quantum machine learning's loss minimization through cross entropy is affected by measurement outcomes.
method Defined quantum cross entropy, proved its lower bounds, and investigated its relation to quantum fidelity and likelihood.
result Quantum cross entropy is lower-bounded by negative log-likelihood when derived from quantum data, but measurement outcomes can cause loss.
Transformer pre-training improves stock return prediction accuracy.
problem Improving stock price prediction accuracy for better investment decisions.
method Pre-trained transformer models on TSX index, fine-tuned for individual stocks, compared to LSTM and XGBoost.
result Transformer model achieved lower mean squared error than benchmarks.
Separating a singing voice from its music accompaniment remains an important challenge in the field of music information retrieval. We present a unique neural network approach inspired by a technique that has revolutionized the field of vision: pixel-wise image classification, which we combine with cross entropy loss a…
State-of-the-art neural networks are vulnerable to adversarial examples; they can easily misclassify inputs that are imperceptibly different than their training and test data. In this work, we establish that the use of cross-entropy loss function and the low-rank features of the training data have responsibility for th…
Cross-entropy loss linked to metric learning, outperforming complex pairwise losses.
problem Improving metric learning performance without complex optimization schemes.
method Theoretical analysis linking cross-entropy to pairwise losses, showing cross-entropy as an upper bound and equivalent to mutual information maximization.
result Minimizing cross-entropy is equivalent to maximizing mutual information, leading to state-of-the-art performance.
Denoising autoencoders (DAEs) are powerful deep learning models used for feature extraction, data generation and network pre-training. DAEs consist of an encoder and decoder which may be trained simultaneously to minimise a loss (function) between an input and the reconstruction of a corrupted version of the input. The…
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
problem Lack of theoretical relationship between softmax cross-entropy and negative sampling loss functions in knowledge graph embedding.
method Used Bregman divergence to provide a unified interpretation of the two loss functions.
result Theoretical findings for fair comparison of softmax cross-entropy and negative sampling are derived.
We translate the problem of calculating the entropy of a set of binary configurations/signals into a sequence of supervised classification tasks. Subsequently, one can use virtually any machine learning classification algorithm for computing entropy. This procedure can be used to compute entropy, and consequently the f…
PAC-Bayesian bounds for MLPs with cross entropy loss validated.
problem Generalization bounds for MLPs with cross entropy loss.
method Introduced probabilistic explanations and proved PAC-Bayesian bounds using ELBO.
result MLPs with cross entropy loss inherently guarantee PAC-Bayesian generalization bounds.
New loss functions improve extreme classification with missing labels.
problem Large number of infrequent labels and missing labels in XMC.
method Derive unbiased loss functions for XMC, incorporating them into existing algorithms.
result Significant improvement in extreme classification performance (up to 20%) over existing methods.
In this paper, we propose majority voting neural networks for sparse signal recovery in binary compressed sensing. The majority voting neural network is composed of several independently trained feedforward neural networks employing the sigmoid function as an activation function. Our empirical study shows that a choice…
Recently the deep learning techniques have achieved success in multi-label classification due to its automatic representation learning ability and the end-to-end learning framework. Existing deep neural networks in multi-label classification can be divided into two kinds: binary relevance neural network (BRNN) and thre…
Unified approach optimizes neural network training for various metrics.
problem Training and evaluation of neural network binary classifiers often use different metrics.
method Combines differentiable approximation and probabilistic soft sets.
result Effective in optimizing for metrics like F1-Score across various domains.
Study Transformer layers under cross-entropy training using mean field control.
problem Understanding the behavior of Transformer layers in cross-entropy training.
method Continuous-depth mean field control analysis, treating depth as time and layer parameters as controls.
result Derivation of a Pontryagin condition for the limiting population problem, involving the softmax residual.
New method detects changes by maximizing cross-entropy, outperforming existing techniques.
problem Detecting abrupt changes in data streams without labeled examples.
method Maximizes cross-entropy between segments to find change points, using dynamic programming.
result Outperforms three state-of-the-art approaches on challenging datasets.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Square loss performs comparably or better than cross-entropy in neural architectures for various tasks.
problem The superiority of cross-entropy loss over square loss in classification tasks is debated.
method Comparison of several neural architectures on NLP, ASR, and computer vision datasets using both loss functions.
result Square loss often produces better results in the majority of tasks, especially in NLP and ASR.
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First H-consistency bounds for comp-sum losses and smooth adversarial comp-sum losses. Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
Entropy measures geodesic flow complexity.
problem Measuring complexity of geodesic flows on manifolds.
method Introduced barcode entropy to measure exponential growth rate of not-too-short bars in Morse-theoretic barcodes.
result Barcode entropy bounds topological entropy and vice versa.
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from co…
Cryptocurrencies are ranked for efficiency using a new Complexity-Entropy Plane.
problem Evaluating the efficiency of cryptocurrencies using traditional financial metrics.
method Developed a Binary Complexity-Entropy Plane (BiCEP) to analyze daily price fluctuations of major cryptocurrencies.
result Only Shiba Inu (SHIB) is significantly inefficient, while most cryptocurrencies operate in close-to-efficient conditions.
Proposes a method to solve deep neural networks' local minimum problem.
problem Local minimum problem in deep neural networks training.
method Transforms cross-entropy loss into risk-averse error criterion, adjusts RSI, and uses convexity region.
result Trained deep learning machine is expected to be inside a global minimum's attraction basin.
Paper proposes a new loss function for conditional models using soft targets.
problem Improving generalization performance of deep neural networks on supervised classification tasks.
method Introduces a new loss function compatible with soft targets, based on noise contrastive estimation.
result Soft target InfoNCE loss performs on par with cross-entropy baselines and outperforms other losses.
We propose new methods for Support Vector Machines (SVMs) using tree architecture for multi-class classi- fication. In each node of the tree, we select an appropriate binary classifier using entropy and generalization error estimation, then group the examples into positive and negative classes based on the selected cla…
This work compares and evaluates various sampling methods for neural language models.
problem Lack of systematic comparison and myths about sampling methods.
method Monte Carlo sampling, importance sampling, compensated partial summation, noise contrastive estimation.
result All sampling methods can perform equally well if posterior probabilities are corrected.
In many real-world prediction tasks, class labels include information about the relative ordering between labels, which is not captured by commonly-used loss functions such as multi-category cross-entropy. Recently, the deep learning community adopted ordinal regression frameworks to take such ordering information into…
This paper presents novel mixed-type Bayesian optimization (BO) algorithms to accelerate the optimization of a target objective function by exploiting correlated auxiliary information of binary type that can be more cheaply obtained, such as in policy search for reinforcement learning and hyperparameter tuning of machi…
Convolution Neural Networks (CNN) have recently achieved state-of-the art performance on handwritten Chinese character recognition (HCCR). However, most of CNN models employ the SoftMax activation function and minimize cross entropy loss, which may cause loss of inter-class information. To cope with this problem, we pr…
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing…
To model categorical response variables given their covariates, we propose a permuted and augmented stick-breaking (paSB) construction that one-to-one maps the observed categories to randomly permuted latent sticks. This new construction transforms multinomial regression into regression analysis of stick-specific binar…
Study of focal-entropy for class-imbalanced classification.
problem Understanding the focal-loss in class-imbalanced settings.
method Distributional viewpoint and information-theoretic analysis of focal-entropy.
result Focal-entropy minimizer exists and departs from data distribution.
Structured entropy improves classification performance on structured targets.
problem Cross-entropy loss fails to account for target variable structure.
method Proposes structured entropy, a generalization of entropy using random partitions.
result Structured cross-entropy loss yields better results on classification problems with known structure.
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
Study proposes a differentiable surrogate loss function for optimizing Fβ score in binary classification with imbalanced data.
problem Non-differentiability of Fβ score makes it unsuitable for optimization by gradient-based learning. method Investigated relationship between Fβ score and loss functions, proposed a differentiable surrogate loss function. result Gradient paths of the proposed surrogate Fβ loss function approximate the gradient paths of the Fβ score. Study on bit threads and their locking properties in holographic spacetimes.
problem Understanding the conditions under which regions can be locked in holographic spacetimes.
method Investigation of different density bounds and their implications on the locking of regions.
result Non-crossing regions can be locked under the most stringent bound, but crossing regions cannot.