Paper introduces hierarchical softmax for global hierarchical classification tasks.
problem Improving classification accuracy in tasks with class hierarchies.
method Global hierarchical neural networks using hierarchical softmax.
result Hierarchical softmax outperforms regular softmax in multiple datasets.
Hierarchical Softmax approximates class probabilities for large datasets efficiently.
problem Computational inefficiency of Softmax for large-scale classification tasks.
method Used Hierarchical Softmax to approximate class probabilities efficiently.
result Hierarchical Softmax performance degrades as the number of classes increases.
Improved HMoE models using Laplace gating function enhance expert specialization and performance.
problem Improving performance of hierarchical mixture of experts models.
method Used Laplace gating function instead of Softmax in hierarchical mixture of experts models.
result Laplace gating function accelerates expert convergence and enhances specialization.
Extends deep learning for hierarchical data to improve classification accuracy.
problem Classification with costly features in hierarchical data.
method Extended deep reinforcement learning with hierarchical deep sets and softmax.
result Superior performance on seven datasets, including malicious web domain classification.
Probabilistic label trees improve XMLC by organizing labels hierarchically.
problem Efficiently tagging instances with a small subset of relevant labels from a large pool.
method Introduce and analyze probabilistic label trees (PLTs) as a generalization of hierarchical softmax for multi-label problems.
result PLTs are consistent for various performance metrics and can be trained online without prior knowledge.
Loss assigns examples to classes and superclasses in hierarchical data.
problem Learning from hierarchical classification problems with known class hierarchy.
method Introduces a loss function that considers the hierarchy of classes, aiming for consistent classification across different granularities.
result Improves accuracy and reduces coarse errors in classification compared to cross-entropy loss.
This work improves metric learning models by incorporating class hierarchies.
problem Class hierarchies are often ignored in classification-based metric learning models.
method Trained softmax classifier and metric learning models with predefined class hierarchies.
result ProxyDR model performs better in hierarchical inference and hierarchy-informed performance.
The paper analyzes convergence rates of softmax gating in MoE models.
problem The effectiveness and scalability of machine learning models using MoE.
method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.
result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.
Efficient algorithm for evaluating hierarchical classification methods at multiple operating points.
problem Evaluating hierarchical classification methods at multiple operating points.
method Efficient algorithm to produce operating characteristic curves for any method that assigns scores to every class in the hierarchy.
result Top-down classifiers are dominated by a naive flat softmax classifier across the entire operating range.
A new catnat function improves gradient descent for categorical variables.
problem Gradient descent challenges with discrete latent categorical variables.
method Replaced softmax with catnat, a hierarchical binary split function.
result Catnat function offers significant advantages in gradient descent.
Transformers handle long documents better with hierarchical segmentation and recurrent layers.
problem Applying BERT to long documents like call transcripts.
method Hierarchical segmentation, recurrent layers, and softmax activation.
result Significant improvement in customer call satisfaction prediction and topic classification tasks.
Extreme multi-label classification (XMLC) is a problem of tagging an instance with a small subset of relevant labels chosen from an extremely large pool of possible labels. Large label spaces can be efficiently handled by organizing labels as a tree, like in the hierarchical softmax (HSM) approach commonly used for mul…
Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unc…
Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which …
Paper proposes a new hierarchical attention mechanism for multi-scale data.
problem Challenges in applying neural attention to multi-scale, multi-modal data.
method Developed a mathematical framework for multi-modal, multi-scale data and derived optimal neural attention mechanics.
result Proposed hierarchical attention mechanism improves transformer performance in multi-scale, multi-modal settings.
Develops RF-softmax for faster training with softmax cross entropy.
problem High computational cost of training with softmax cross entropy.
method Random Fourier Features for efficient sampling from approximate softmax distribution.
result RF-softmax provides low bias in estimating both softmax distribution and its gradient.
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
problem Inherent uncertainty in logistic-softmax leads to suboptimal performance in meta-learning.
method Redesigns logistic-softmax likelihood with a temperature parameter for better control of prior confidence.
result Achieves well-calibrated uncertainty estimates and comparable/superior performance on benchmark datasets.
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
problem Long-tailed distribution mismatch between training and testing data.
method Balanced Meta-Softmax, an unbiased extension of Softmax, using a Meta Sampler.
result Balanced Meta-Softmax outperforms state-of-the-art solutions on visual recognition and instance segmentation.
DS-Softmax speeds up softmax inference by learning sparse experts.
problem Expensive softmax computations for large output classes.
method Sparse mixture of sparse experts for efficient top-k class retrieval.
result Significant computation reductions achieved at no performance loss.
Softmax emerges naturally in neural networks as a measure of conditional mutual information.
problem The artificial nature of softmax in neural networks.
method Information-theoretic perspective to derive log-softmax and evaluate conditional mutual information.
result Training deterministic neural networks through log-softmax maximises conditional mutual information.
Binary testing for softmax models requires many samples, similar to leverage score models.
problem Binary hypothesis testing for softmax models and leverage score models.
method Analyzing sample complexity and drawing analogies between models.
result Sample complexity is asymptotically \(O(ε^{-2})\), where \(ε\) is the distance between model parameters.
The paper investigates polynomial alternatives to softmax in transformer models.
problem The effectiveness of softmax attention in transformers is questioned.
method The authors explore polynomial activations as alternatives to softmax, focusing on their ability to regularize the attention matrix.
result Certain polynomials can serve as effective substitutes for softmax in transformer applications, achieving strong performance.
Softmax temperature influences model representation rank and performance.
problem Understanding and optimizing softmax function's impact on model representations.
method Investigated softmax function's role in deep neural networks, introduced rank deficit bias.
result Softmax temperature affects model representation rank and can improve performance.
Improved classifier accuracy by using more of the class-specific structure in trained models.
problem Softmax ignores valuable information encoded in the full array of class response distributions.
method Developed a hybrid classifier (Softmax-Pooling Hybrid, SPH) that uses Softmax on high-scoring samples and a log-likelihood method on low-scoring samples. result Reduces test set error by 6% to 23% using the exact same trained model.
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a gene…
Paper proposes a new softmax loss for better performance in Positive and Unlabeled data tasks.
problem Current softmax losses and sampling schemes have drawbacks in Positive and Unlabeled learning.
method Proposes Relaxed Softmax (RS) loss and a new negative sampling scheme.
result New training objective drives uplifts in performance on textual and recommendation datasets.
Softmax confidence misrepresents uncertainty in neural networks.
problem Neural networks fail to increase uncertainty on out-of-distribution data.
method Investigates two implicit biases in softmax confidence.
result Softmax confidence correlates with epistemic uncertainty due to decision boundary structure and deep network filtering.
Unified framework for studying softmax attention under large prompts.
problem Challenges in theoretical analysis of softmax attention.
method Measure-based framework for finite and infinite prompts.
result Softmax attention converges to linear attention in the large-prompt regime.
In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood pri…
Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper …
Transformers use ReLUs to approximate softmax efficiently.
problem Analyzing resource usage in softmax transformer models.
method Translating ReLU approximation results to softmax attention mechanisms.
result Economic resource bounds for softmax attention mechanisms.
Paper shows softmax output misleads in evaluating adversarial example strength.
problem Softmax output misleads in evaluating adversarial example strength.
method Demonstrates how adversarial examples can exploit softmax properties.
result Softmax output is a poor indicator of adversarial example strength.
Improved SincNet for better speaker recognition.
problem Speaker recognition challenges and the need for better deep learning models.
method Proposes AM-SincNet, a SincNet-based model with an improved AM-Softmax layer.
result Improved speaker recognition performance, achieving a 40% Frame Error Rate reduction.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
DBS improves convergence in reinforcement learning.
problem Softmax operator convergence issues in reinforcement learning.
method Dynamic Boltzmann Softmax (DBS) updates value function.
result DBS enables better value function estimation and convergence.
A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a sing…
The paper proves neural networks with ReLU and softmax can approximate any function.
problem Approximating functions and class labels in neural networks.
method Extended universal approximator theory to neural networks with ReLU and softmax.
result Neural networks with ReLU and softmax can approximate any function and class labels.
Proposes learnable monotonic functions to improve softmax's limitations.
problem Softmax's limited representational capacity in large output vocabularies.
method Learn parametric monotonic functions on logits.
result Improves quality metrics over traditional Linear-Softmax in language models.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
problem Lack of theoretical relationship between softmax cross-entropy and negative sampling loss functions in knowledge graph embedding.
method Used Bregman divergence to provide a unified interpretation of the two loss functions.
result Theoretical findings for fair comparison of softmax cross-entropy and negative sampling are derived.
New L2 regularization improves softmax MAB performance.
problem Improving softmax MAB performance with vanishing regularization.
method L2 regularization with vanishing parameter analyzed and proven convergent.
result Vanishing L2 regularization makes softmax MAB more numerically advantageous.
Gradient flow in softmax models tends to produce low-entropy outputs.
problem Understanding the training dynamics of softmax-based models.
method Analysis of gradient flow dynamics in the value-softmax model.
result Gradient flow drives optimization towards low-entropy solutions.
This note considers softmax parameter estimation when little/no labeled training data is available, but a priori information about the relative geometry of class label log-odds boundaries is available. It is shown that `data-free' softmax model synthesis corresponds to solving a linear system of parameter equations, wh…
Softmax PG methods can take extremely long to converge, even with exact gradients.
problem Softmax policy gradient methods can take an impractically long time to converge in reinforcement learning.
method Softmax policy gradient methods with exact gradient computation.
result Softmax PG methods can take exponential number of iterations to converge, even with optimal initialization.
Softmax Bellman operator improves Q-function performance in RL despite sub-optimality.
problem Softmax Bellman operator's impact on value functions in RL is problematic.
method Revisited theoretical properties of softmax Bellman operator, proving convergence and overestimation reduction.
result Softmax Bellman operator leads to superior policies in practice, even outperforming double Q-learning.
WEINCE improves contrastive learning by correcting softmax biases.
problem Softmax in InfoNCE can lead to misaligned statistical assumptions in contrastive learning.
method WEINCE uses anchor-wise online batch statistics to blend softmax logits with an endpoint shortfall correction.
result WEINCE yields consistent improvements in frozen-feature evaluation across five vision benchmarks.
Density-Softmax improves uncertainty estimation and robustness without sampling, reducing model size and latency.
problem Sampling-based uncertainty estimation methods suffer from large model size and high latency.
method Combines a Lipschitz-constrained feature extractor with the softmax layer to create a sampling-free deterministic framework.
result Density-Softmax reduces over-confidence under distribution shifts and achieves competitive results in uncertainty and robustness.