DS-Softmax speeds up softmax inference by learning sparse experts.
problem Expensive softmax computations for large output classes.
method Sparse mixture of sparse experts for efficient top-k class retrieval.
result Significant computation reductions achieved at no performance loss.
Proposes a new method to approximate Gaussian inference in classification tasks.
problem Uncertainty quantification in classification tasks using softmax functions.
method Develops a new formalism to approximate Gaussian distributions over logit space and proposes using element-wise normCDF or sigmoid instead of softmax.
result Improves uncertainty quantification compared to softmax Monte Carlo sampling.
Neural language models have been widely used in various NLP tasks, including machine translation, next word prediction and conversational agents. However, it is challenging to deploy these models on mobile devices due to their slow prediction speed, where the bottleneck is to compute top candidates in the softmax layer…
We study the tradeoff between computational effort and classification accuracy in a cascade of deep neural networks. During inference, the user sets the acceptable accuracy degradation which then automatically determines confidence thresholds for the intermediate classifiers. As soon as the confidence threshold is met,…
A new loss function improves neural networks' out-of-distribution detection without side effects.
problem Neural networks struggle with out-of-distribution detection due to SoftMax loss issues.
method Proposes IsoMax loss replacing SoftMax loss, maintaining high entropy and fast inferences.
result Significantly improves neural networks' out-of-distribution detection performance.
New CNN approach reduces overconfidence in object classification predictions.
problem Overconfident predictions from deep models, especially SoftMax layer.
method Introduces CNN probabilistic approach using Logit layer for Bayesian inference.
result Proposed approach shows promising performance compared to SoftMax.
The softmax representation of probabilities for categorical variables plays a prominent role in modern machine learning with numerous applications in areas such as large scale classification, neural language modeling and recommendation systems. However, softmax estimation is very expensive for large scale inference bec…
Efficient multi-class classification with well-calibrated uncertainty.
problem Trade-off between uncertainty calibration and speed in multi-class Gaussian process classification.
method Proposes a new likelihood function leading to a conditionally conjugate model with efficient variational inference.
result Up to two orders faster than state-of-the-art methods with well-calibrated uncertainty estimates.
New loss improves OOD detection without extra data or tuning.
problem Improving OOD detection without additional data or tuning.
method Proposed IsoMax loss and entropic score to replace SoftMax loss.
result Training with IsoMax loss significantly improves OOD detection performance.
We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss func…
Transformers learn to cluster Gaussian mixtures as well as the EM algorithm.
problem Learning guarantees of Transformers in multi-class clustering of Gaussian mixtures.
method Developed a theory connecting Transformer's Softmax Attention layers to the EM algorithm's workflow.
result Transformers achieve minimax optimal rate for clustering Gaussian mixtures with sufficient training samples and initialization.
DRNets dynamically route instances to efficient transformations.
problem High inference costs due to static model capacity.
method Dynamic Routing Networks (DRNets) with RouterNets for branch selection.
result DRNets reduce inference costs with comparable performance.
A new method for uncertainty estimation in neural networks using Gaussian-softmax integration.
problem Quantifying uncertainty in neural network predictions.
method Proposes a single-model approach integrating Gaussian distribution with softmax outputs, using mean-field approximation.
result Competitive performance on uncertainty estimation tasks and outperforms many methods on out-of-distribution detection.
With an eye towards human-centered automation, we contribute to the development of a systematic means to infer features of human decision-making from behavioral data. Motivated by the common use of softmax selection in models of human decision-making, we study the maximum likelihood parameter estimation problem for sof…
We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This parameter tying enables an efficient pretraining algorithm and a …
The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for rep…
Develops RF-softmax for faster training with softmax cross entropy.
problem High computational cost of training with softmax cross entropy.
method Random Fourier Features for efficient sampling from approximate softmax distribution.
result RF-softmax provides low bias in estimating both softmax distribution and its gradient.
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
A method to automatically choose feature dimensions in linear attention for better approximation quality.
problem Choosing the feature dimension in linear attention to balance quality and efficiency.
method Statistical degrees of freedom for determining feature dimension, layer-wise training strategy.
result Our method achieves smaller approximation error compared to fixed dimensions and improves model performance.
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
problem Inherent uncertainty in logistic-softmax leads to suboptimal performance in meta-learning.
method Redesigns logistic-softmax likelihood with a temperature parameter for better control of prior confidence.
result Achieves well-calibrated uncertainty estimates and comparable/superior performance on benchmark datasets.
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
problem Long-tailed distribution mismatch between training and testing data.
method Balanced Meta-Softmax, an unbiased extension of Softmax, using a Meta Sampler.
result Balanced Meta-Softmax outperforms state-of-the-art solutions on visual recognition and instance segmentation.
Paper introduces hierarchical softmax for global hierarchical classification tasks.
problem Improving classification accuracy in tasks with class hierarchies.
method Global hierarchical neural networks using hierarchical softmax.
result Hierarchical softmax outperforms regular softmax in multiple datasets.
Typically, Softmax is used in the final layer of a neural network to get a probability distribution for output classes. But the main problem with Softmax is that it is computationally expensive for large scale data sets with large number of possible outputs. To approximate class probability efficiently on such large sc…
Softmax emerges naturally in neural networks as a measure of conditional mutual information.
problem The artificial nature of softmax in neural networks.
method Information-theoretic perspective to derive log-softmax and evaluate conditional mutual information.
result Training deterministic neural networks through log-softmax maximises conditional mutual information.
Binary testing for softmax models requires many samples, similar to leverage score models.
problem Binary hypothesis testing for softmax models and leverage score models.
method Analyzing sample complexity and drawing analogies between models.
result Sample complexity is asymptotically \(O(ε^{-2})\), where \(ε\) is the distance between model parameters.
The paper investigates polynomial alternatives to softmax in transformer models.
problem The effectiveness of softmax attention in transformers is questioned.
method The authors explore polynomial activations as alternatives to softmax, focusing on their ability to regularize the attention matrix.
result Certain polynomials can serve as effective substitutes for softmax in transformer applications, achieving strong performance.
Softmax temperature influences model representation rank and performance.
problem Understanding and optimizing softmax function's impact on model representations.
method Investigated softmax function's role in deep neural networks, introduced rank deficit bias.
result Softmax temperature affects model representation rank and can improve performance.
Improved classifier accuracy by using more of the class-specific structure in trained models.
problem Softmax ignores valuable information encoded in the full array of class response distributions.
method Developed a hybrid classifier (Softmax-Pooling Hybrid, SPH) that uses Softmax on high-scoring samples and a log-likelihood method on low-scoring samples. result Reduces test set error by 6% to 23% using the exact same trained model.
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a gene…
Paper proposes a new softmax loss for better performance in Positive and Unlabeled data tasks.
problem Current softmax losses and sampling schemes have drawbacks in Positive and Unlabeled learning.
method Proposes Relaxed Softmax (RS) loss and a new negative sampling scheme.
result New training objective drives uplifts in performance on textual and recommendation datasets.
Softmax confidence misrepresents uncertainty in neural networks.
problem Neural networks fail to increase uncertainty on out-of-distribution data.
method Investigates two implicit biases in softmax confidence.
result Softmax confidence correlates with epistemic uncertainty due to decision boundary structure and deep network filtering.
Unified framework for studying softmax attention under large prompts.
problem Challenges in theoretical analysis of softmax attention.
method Measure-based framework for finite and infinite prompts.
result Softmax attention converges to linear attention in the large-prompt regime.
In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood pri…
Simplified plug-in loss approximates EDL for reliable uncertainty estimation.
problem Efficient and reliable uncertainty estimation in real-world sensor-based learning systems.
method Approximate Dirichlet expected objectives with plug-in losses evaluated at the Dirichlet mean.
result Plug-in losses provide comparable predictive accuracy and selective prediction performance to classical EDL, while being simpler to implement.
Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper …
Transformers use ReLUs to approximate softmax efficiently.
problem Analyzing resource usage in softmax transformer models.
method Translating ReLU approximation results to softmax attention mechanisms.
result Economic resource bounds for softmax attention mechanisms.
Improved SincNet for better speaker recognition.
problem Speaker recognition challenges and the need for better deep learning models.
method Proposes AM-SincNet, a SincNet-based model with an improved AM-Softmax layer.
result Improved speaker recognition performance, achieving a 40% Frame Error Rate reduction.
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
DBS improves convergence in reinforcement learning.
problem Softmax operator convergence issues in reinforcement learning.
method Dynamic Boltzmann Softmax (DBS) updates value function.
result DBS enables better value function estimation and convergence.
A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a sing…
The paper proves neural networks with ReLU and softmax can approximate any function.
problem Approximating functions and class labels in neural networks.
method Extended universal approximator theory to neural networks with ReLU and softmax.
result Neural networks with ReLU and softmax can approximate any function and class labels.
Proposes learnable monotonic functions to improve softmax's limitations.
problem Softmax's limited representational capacity in large output vocabularies.
method Learn parametric monotonic functions on logits.
result Improves quality metrics over traditional Linear-Softmax in language models.
Most of the parameters in large vocabulary models are used in embedding layer to map categorical features to vectors and in softmax layer for classification weights. This is a bottle-neck in memory constraint on-device training applications like federated learning and on-device inference applications like automatic spe…
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value …
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
problem Lack of theoretical relationship between softmax cross-entropy and negative sampling loss functions in knowledge graph embedding.
method Used Bregman divergence to provide a unified interpretation of the two loss functions.
result Theoretical findings for fair comparison of softmax cross-entropy and negative sampling are derived.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
New L2 regularization improves softmax MAB performance.
problem Improving softmax MAB performance with vanishing regularization.
method L2 regularization with vanishing parameter analyzed and proven convergent.
result Vanishing L2 regularization makes softmax MAB more numerically advantageous.