A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
The paper investigates polynomial alternatives to softmax in transformer models.
In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood pri…
Paper extends sparse alternatives to softmax for continuous domains, enabling efficient attention mechanisms.
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a sing…
We present ADMM-Softmax, an alternating direction method of multipliers (ADMM) for solving multinomial logistic regression (MLR) problems. Our method is geared toward supervised classification tasks with many examples and features. It decouples the nonlinear optimization problem in MLR into three steps that can be solv…
Neural networks utilize the softmax as a building block in classification tasks, which contains an overconfidence problem and lacks an uncertainty representation ability. As a Bayesian alternative to the softmax, we consider a random variable of a categorical probability over class labels. In this framework, the prior …
Evidential Softmax preserves multimodality in sparse probability distributions for generative models.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
This paper improves MADDPG's performance in discrete grid-world scenarios.
A new catnat function improves gradient descent for categorical variables.
We formally prove the connection between k-means clustering and the predictions of neural networks based on the softmax activation layer. In existing work, this connection has been analyzed empirically, but it has never before been mathematically derived. The softmax function partitions the transformed input space into…
The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…
A new topology design improves zero-shot classification performance in contrastive learning.
Transformers model contextual relations using probabilistic measures, revealing their expressive power.
Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unc…
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
Unified framework for sequence models using test-time regression.
The computational cost of training with softmax cross entropy loss grows linearly with the number of classes. For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as…
Converting an n-dimensional vector to a probability distribution over n objects is a commonly used component in many machine learning tasks like multiclass classification, multilabel classification, attention mechanisms etc. For this, several probability mapping functions have been proposed and employed in literature s…
This note is concerned with accurate and computationally efficient approximations of moments of Gaussian random variables passed through sigmoid or softmax mappings. These approximations are semi-analytical (i.e. they involve the numerical adjustment of parametric forms) and highly accurate (they yield 5% error at most…
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
Improves uncertainty estimation and OOD detection in neural networks.
PARC uses piecewise linear predictors for regression and classification.
Paper introduces hierarchical softmax for global hierarchical classification tasks.
Typically, Softmax is used in the final layer of a neural network to get a probability distribution for output classes. But the main problem with Softmax is that it is computationally expensive for large scale data sets with large number of possible outputs. To approximate class probability efficiently on such large sc…
New method uses temperature to control sparse MoE convergence rates.
This research improves neural network uncertainty estimates and reliability.
Binary testing for softmax models requires many samples, similar to leverage score models.
Softmax temperature influences model representation rank and performance.
Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an ar…
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a gene…
Despite great popularity of applying softmax to map the non-normalised outputs of a neural network to a probability distribution over predicting classes, this normalised exponential transformation still seems to be artificial. A theoretic framework that incorporates softmax as an intrinsic component is still lacking. I…
Softmax confidence misrepresents uncertainty in neural networks.
Computations for the softmax function are significantly expensive when the number of output classes is large. In this paper, we present a novel softmax inference speedup method, Doubly Sparse Softmax (DS-Softmax), that leverages sparse mixture of sparse experts to efficiently retrieve top-k classes. Different from most…
Efficiently compress pretrained models using RSI for improved predictive accuracy.
Unified framework for studying softmax attention under large prompts.
Interpolates between SPG and NeuRD with Capped Implicit Exploration.
Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper …
Transformers use ReLUs to approximate softmax efficiently.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
Speaker Recognition is a challenging task with essential applications such as authentication, automation, and security. The SincNet is a new deep learning based model which has produced promising results to tackle the mentioned task. To train deep learning systems, the loss function is essential to the network performa…
This paper develops sparse alternatives to continuous distributions, including new types of Gaussians and attention mechanisms.
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
Introduces gradient decay in Softmax for better generalization.
New L2 regularization improves softmax MAB performance.
Gradient flow in softmax models tends to produce low-entropy outputs.