A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Unified framework for gradient estimation in combinatorial spaces.
Paper proposes a new softmax loss for better performance in Positive and Unlabeled data tasks.
A new method reparameterizes Gaussian noise for better flexibility and performance.
Paper proposes a new method for predicting drug interactions using adversarial autoencoders.
A new method for optimizing regression problems with ReLU units converges.
In many applications we seek to maximize an expectation with respect to a distribution over discrete variables. Estimating gradients of such objectives with respect to the distribution parameters is a challenging problem. We analyze existing solutions including finite-difference (FD) estimators and continuous relaxatio…
A new method for categorical variational inference using discrete normalizing flows.
This paper improves MADDPG's performance in discrete grid-world scenarios.
Many machine learning tasks require sampling a subset of items from a collection based on a parameterized distribution. The Gumbel-softmax trick can be used to sample a single item, and allows for low-variance reparameterized gradients with respect to the parameters of the underlying distribution. However, stochastic o…
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
Enhances neural architecture search efficiency and prevents performance collapse.
The computational cost of training with softmax cross entropy loss grows linearly with the number of classes. For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as…
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
Reparameterization of variational auto-encoders with continuous random variables is an effective method for reducing the variance of their gradient estimates. In the discrete case, one can perform reparametrization using the Gumbel-Max trick, but the resulting objective relies on an operation and is non-dif…
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
Paper introduces hierarchical softmax for global hierarchical classification tasks.
Typically, Softmax is used in the final layer of a neural network to get a probability distribution for output classes. But the main problem with Softmax is that it is computationally expensive for large scale data sets with large number of possible outputs. To approximate class probability efficiently on such large sc…
Softmax emerges naturally in neural networks as a measure of conditional mutual information.
Binary testing for softmax models requires many samples, similar to leverage score models.
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary lat…
The paper investigates polynomial alternatives to softmax in transformer models.
Softmax temperature influences model representation rank and performance.
Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an ar…
Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a gene…
Modified EAT method improves Poisson gradient estimation.
Softmax confidence misrepresents uncertainty in neural networks.
Computations for the softmax function are significantly expensive when the number of output classes is large. In this paper, we present a novel softmax inference speedup method, Doubly Sparse Softmax (DS-Softmax), that leverages sparse mixture of sparse experts to efficiently retrieve top-k classes. Different from most…
Unified framework for studying softmax attention under large prompts.
Topic sparsity refers to the observation that individual documents usually focus on several salient topics instead of covering a wide variety of topics, and a real topic adopts a narrow range of terms instead of a wide coverage of the vocabulary. Understanding this topic sparsity is especially important for analyzing u…
In a multi-class classification problem, it is standard to model the output of a neural network as a categorical distribution conditioned on the inputs. The output must therefore be positive and sum to one, which is traditionally enforced by a softmax. This probabilistic mapping allows to use the maximum likelihood pri…
Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper …
Transformers use ReLUs to approximate softmax efficiently.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a sing…
Speaker Recognition is a challenging task with essential applications such as authentication, automation, and security. The SincNet is a new deep learning based model which has produced promising results to tackle the mentioned task. To train deep learning systems, the loss function is essential to the network performa…
Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable…
The paper proves neural networks with ReLU and softmax can approximate any function.
Unified interpretation of softmax cross-entropy and negative sampling for knowledge graph embedding.
Introduces gradient decay in Softmax for better generalization.
New L2 regularization improves softmax MAB performance.
Gradient flow in softmax models tends to produce low-entropy outputs.
Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the non-expansion property, and its direct use may fail to converge even in value iteration. In this pape…
This note considers softmax parameter estimation when little/no labeled training data is available, but a priori information about the relative geometry of class label log-odds boundaries is available. It is shown that `data-free' softmax model synthesis corresponds to solving a linear system of parameter equations, wh…
Softmax PG methods can take extremely long to converge, even with exact gradients.
WEINCE improves contrastive learning by correcting softmax biases.
Density-Softmax improves uncertainty estimation and robustness without sampling, reducing model size and latency.
Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using…
Softmax is found ineffective for NL block, leading to improved performance.