A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Binary testing for softmax models requires many samples, similar to leverage score models.
Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an ar…
Typically, Softmax is used in the final layer of a neural network to get a probability distribution for output classes. But the main problem with Softmax is that it is computationally expensive for large scale data sets with large number of possible outputs. To approximate class probability efficiently on such large sc…
WEINCE improves contrastive learning by correcting softmax biases.
Quaternion self-attention reduces computational cost and improves performance.
This research improves neural network uncertainty estimates and reliability.
Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unc…
Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are de…
Improved logistic MoE with sigmoid gate shows better sample efficiency.
BoC probe assesses neural network confidence coherence, revealing architecture-specific uncertainty.
Out-of-distribution (OOD) detection approaches usually present special requirements (e.g., hyperparameter validation, collection of outlier data) and produce side effects (e.g., classification accuracy drop, slower energy-inefficient inferences). We argue that these issues are a consequence of the SoftMax loss anisotro…
A new DR formulation improves metric learning for faster and more stable performance.
New conformal prediction methods for long-tailed classification problems.
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
Attention temperature improves robustness of ICL in high-dimensional settings.
The computational cost of training with softmax cross entropy loss grows linearly with the number of classes. For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as…
Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the exist…
CNNs improve wind speed forecasts in the Netherlands.
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
Paper introduces hierarchical softmax for global hierarchical classification tasks.
Unified framework for critical scaling of inverse temperature in self-attention.
The paper investigates polynomial alternatives to softmax in transformer models.
New research shows input-gradients can be manipulated without changing model's core function, challenging their use for model interpretation.
Softmax temperature influences model representation rank and performance.
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a gene…
Despite great popularity of applying softmax to map the non-normalised outputs of a neural network to a probability distribution over predicting classes, this normalised exponential transformation still seems to be artificial. A theoretic framework that incorporates softmax as an intrinsic component is still lacking. I…
Softmax confidence misrepresents uncertainty in neural networks.
Computations for the softmax function are significantly expensive when the number of output classes is large. In this paper, we present a novel softmax inference speedup method, Doubly Sparse Softmax (DS-Softmax), that leverages sparse mixture of sparse experts to efficiently retrieve top-k classes. Different from most…
Unified framework for studying softmax attention under large prompts.
In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood pri…
Automated multi-task learning algorithm that optimizes network topology.
Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper …
Transformers use ReLUs to approximate softmax efficiently.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a sing…
Speaker Recognition is a challenging task with essential applications such as authentication, automation, and security. The SincNet is a new deep learning based model which has produced promising results to tackle the mentioned task. To train deep learning systems, the loss function is essential to the network performa…
Convolutional neural networks (CNNs) are similar to "ordinary" neural networks in the sense that they are made up of hidden layers consisting of neurons with "learnable" parameters. These neurons receive inputs, performs a dot product, and then follows it with a non-linearity. The whole network expresses the mapping be…
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
Introduces gradient decay in Softmax for better generalization.
New L2 regularization improves softmax MAB performance.
Gradient flow in softmax models tends to produce low-entropy outputs.
Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the non-expansion property, and its direct use may fail to converge even in value iteration. In this pape…
Efficient algorithm for evaluating hierarchical classification methods at multiple operating points.
This note considers softmax parameter estimation when little/no labeled training data is available, but a priori information about the relative geometry of class label log-odds boundaries is available. It is shown that `data-free' softmax model synthesis corresponds to solving a linear system of parameter equations, wh…
Softmax PG methods can take extremely long to converge, even with exact gradients.
Proposes a new method to measure epistemic uncertainty in Bayesian neural networks.
In recent years, the softmax model and its fast approximations have become the de-facto loss functions for deep neural networks when dealing with multi-class prediction. This loss has been extended to language modeling and recommendation, two fields that fall into the framework of learning from Positive and Unlabeled d…