The paper uses attention networks for character-based handwritten text transcription.
problem Handwritten text recognition with improved character-level alignment.
method Attentional encoder-decoder networks trained on character sequences, comparing different activation functions.
result Softmax attention provides more precise character alignment than sigmoid attention.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
Balanced Activation improves object detection performance on long-tailed datasets.
problem Mismatch between training and testing label distributions in object detection.
method Introduces Balanced Activation (Balanced Softmax and Balanced Sigmoid) to address label distribution shift.
result Balanced Activation provides ~3% gain in mAP on LVIS-1.0 compared to state-of-the-art methods.
Proposes a new method to approximate Gaussian inference in classification tasks.
problem Uncertainty quantification in classification tasks using softmax functions.
method Develops a new formalism to approximate Gaussian distributions over logit space and proposes using element-wise normCDF or sigmoid instead of softmax.
result Improves uncertainty quantification compared to softmax Monte Carlo sampling.
Proposes sigsoftmax to overcome the softmax bottleneck in language models.
problem Softmax function acts as a bottleneck in neural network representational capacity.
method Identifies the cause of softmax bottleneck and proposes sigsoftmax as a new activation function.
result Sigsoftmax outperforms softmax in language modeling tasks.
This note is concerned with accurate and computationally efficient approximations of moments of Gaussian random variables passed through sigmoid or softmax mappings. These approximations are semi-analytical (i.e. they involve the numerical adjustment of parametric forms) and highly accurate (they yield 5% error at most…
Paper explains learning property of logistic and softmax losses for balanced and imbalanced class data.
problem Understanding and optimizing loss functions for deep neural networks with class imbalances.
method Analyzing necessary conditions for convergence of logistic and softmax losses in CNNs.
result Proposes a novel reweighted logistic loss function that improves performance over softmax loss.
Improves uncertainty estimation and OOD detection in neural networks.
problem Accurate uncertainty estimation and OOD detection in neural networks.
method Investigates one-vs-all and distance-based logit representations for probabilities.
result One-vs-all formulations improve calibration without additional complexity.
Hierarchical Softmax approximates class probabilities for large datasets efficiently.
problem Computational inefficiency of Softmax for large-scale classification tasks.
method Used Hierarchical Softmax to approximate class probabilities efficiently.
result Hierarchical Softmax performance degrades as the number of classes increases.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
New ACE cost function encourages diversity in neural networks.
problem Training multiple classifiers with controlled diversity.
method Mathematical derivation and gradient control.
result ACE yields better ensemble results than vanilla.
The softmax representation of probabilities for categorical variables plays a prominent role in modern machine learning with numerous applications in areas such as large scale classification, neural language modeling and recommendation systems. However, softmax estimation is very expensive for large scale inference bec…
Investigates least squares estimation in deterministic MoE models.
problem Complexity and difficulty in analyzing MoE models.
method Least squares estimation under deterministic MoE models.
result Rates of estimating strongly identifiable experts are faster than polynomial experts.
Bayesian method improves few-shot classification accuracy.
problem Few-shot classification with small labeled datasets.
method Gaussian process classifier with Pólya-Gamma augmentation and one-vs-each softmax.
result Improved accuracy and uncertainty quantification.
New bounds show transformers need longer training for length generalization.
problem Understanding when transformers can generalize to longer inputs.
method Analyzing different settings of transformers, providing quantitative bounds.
result Transformers need training data longer than previously thought for length generalization.
Python scripts analyze MRAM-based neuromorphic devices' process variation impacts on machine learning accuracy.
problem Impact of process variation on MRAM-based neuromorphic devices' performance in machine learning applications.
method Developed transportable Python scripts to analyze output variation under changes in device dimensions.
result Revealed impacts and limits for processing variation of device fabrication on energy vs. accuracy tradeoffs.
Proposes a new method to improve multiclass probability calibration.
problem Uncalibrated class probabilities in multiclass classifiers leading to over-confidence.
method Dirichlet calibration method applicable to any model class, derived from Dirichlet distributions.
result Improved probabilistic predictions across various datasets and classifiers.
Paper extends understanding of deep network nonlinearities using vector quantization and statistical inference.
problem Limited understanding of deep network nonlinearities, especially non-piecewise affine and non-convex functions.
method Link deterministic max-affine spline operators to probabilistic Gaussian Mixture Models (GMMs) for a broader class of nonlinearities.
result Enforces orthogonality in linear filters can significantly improve deep network performance.
Single deep model detects out-of-distribution data with single forward pass.
problem Detecting out-of-distribution data points in neural networks.
method Deterministic uncertainty quantification (DUQ) using gradient penalty for reliable detection.
result Single model outperforms or matches ensemble methods in out-of-distribution detection.
Morphology in unbalanced languages remains a big challenge in the context of machine translation. In this paper, we propose to de-couple machine translation from morphology generation in order to better deal with the problem. We investigate the morphology simplification with a reasonable trade-off between expected gain…
In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more…
Develops ADMM for deep neural networks with sigmoid activations to avoid saturation and improve approximation.
problem Gradient saturation in deep neural networks with sigmoid activations.
method Introduces sigmoid-ADMM pair for training deep sigmoid nets and proves its convergence.
result ADMM avoids saturation and improves approximation of deep sigmoid nets compared to ReLU nets.
Simple neural net outperforms complex uncertainty methods.
problem Reliable uncertainty estimation from deterministic models.
method A simple baseline using a single softmax neural net with residual connections and spectral normalization.
result Simple neural net outperforms DUQ and SNGP on uncertainty prediction.
Study on learning to defer to multiple experts with consistent surrogates and confidence calibration.
problem Addressing the open problems of consistent surrogates, confidence calibration, and ensembling of experts.
method Derive two consistent surrogates (softmax and OvA) and propose a conformal inference technique for choosing experts.
result The OvA-based loss does not cause mis-calibration propagation, while the softmax-based loss does.
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
New method estimates sigmoids' parameters using gradient estimation.
problem Learning parameters of models with hidden variables.
method Estimate gradients at random points, cluster, use cluster centers as parameter estimates.
result Proven that estimated gradients concentrate around true parameter vectors.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
Develops RF-softmax for faster training with softmax cross entropy.
problem High computational cost of training with softmax cross entropy.
method Random Fourier Features for efficient sampling from approximate softmax distribution.
result RF-softmax provides low bias in estimating both softmax distribution and its gradient.
BoC probe assesses neural network confidence coherence, revealing architecture-specific uncertainty.
problem Poor calibration and OOD detection in neural networks.
method Bag-of-Coins (BoC) probe compares softmax confidence to pairwise dominance probabilities.
result BoC reveals clear ID/OOD separation for some architectures but not others.
Optimal ANN pre-training with SDA reduces handwritten Bengali digit recognition error to 2.34%
problem Optimizing ANN architecture for Bengali handwritten digit recognition
method Pre-training ANN with stacked denoising autoencoder (SDA)
result Minimum validation error of 2.34% on handwritten Bengali dataset
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
problem Inherent uncertainty in logistic-softmax leads to suboptimal performance in meta-learning.
method Redesigns logistic-softmax likelihood with a temperature parameter for better control of prior confidence.
result Achieves well-calibrated uncertainty estimates and comparable/superior performance on benchmark datasets.
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
problem Long-tailed distribution mismatch between training and testing data.
method Balanced Meta-Softmax, an unbiased extension of Softmax, using a Meta Sampler.
result Balanced Meta-Softmax outperforms state-of-the-art solutions on visual recognition and instance segmentation.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Proposes a new LSTM gate structure using bivariate Beta distribution.
problem Inflexibility of sigmoid gates in modeling multi-modality and skewness, and lack of modeling correlation between gates.
method Introduces a bivariate Beta distribution gate structure within LSTM cells.
result Empirically shows higher gradient values and improved model performance.
Paper introduces hierarchical softmax for global hierarchical classification tasks.
problem Improving classification accuracy in tasks with class hierarchies.
method Global hierarchical neural networks using hierarchical softmax.
result Hierarchical softmax outperforms regular softmax in multiple datasets.
DS-Softmax speeds up softmax inference by learning sparse experts.
problem Expensive softmax computations for large output classes.
method Sparse mixture of sparse experts for efficient top-k class retrieval.
result Significant computation reductions achieved at no performance loss.
Softmax emerges naturally in neural networks as a measure of conditional mutual information.
problem The artificial nature of softmax in neural networks.
method Information-theoretic perspective to derive log-softmax and evaluate conditional mutual information.
result Training deterministic neural networks through log-softmax maximises conditional mutual information.
Binary testing for softmax models requires many samples, similar to leverage score models.
problem Binary hypothesis testing for softmax models and leverage score models.
method Analyzing sample complexity and drawing analogies between models.
result Sample complexity is asymptotically \(O(ε^{-2})\), where \(ε\) is the distance between model parameters.
New algorithms make softmax optimization unbiased and scalable.
problem Efficiently computing softmax distributions with large categories.
method Proposed unbiased algorithms for maximizing softmax likelihood.
result Comprehensive outperformance on seven real-world datasets.
Automatic music transcription (AMT) aims to infer a latent symbolic representation of a piece of music (piano-roll), given a corresponding observed audio recording. Transcribing polyphonic music (when multiple notes are played simultaneously) is a challenging problem, due to highly structured overlapping between harmon…
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
The paper investigates polynomial alternatives to softmax in transformer models.
problem The effectiveness of softmax attention in transformers is questioned.
method The authors explore polynomial activations as alternatives to softmax, focusing on their ability to regularize the attention matrix.
result Certain polynomials can serve as effective substitutes for softmax in transformer applications, achieving strong performance.
Softmax temperature influences model representation rank and performance.
problem Understanding and optimizing softmax function's impact on model representations.
method Investigated softmax function's role in deep neural networks, introduced rank deficit bias.
result Softmax temperature affects model representation rank and can improve performance.
Earth observation embeddings can convert discrete biome maps into continuous representations that better capture ecological variation.
problem Biome maps impose categorical boundaries that compress continuous variation in biotic communities.
method Fit a linear classifier on Earth observation embeddings to predict biome labels.
result Continuous biome representation outperforms discrete biome labels for predicting species occurrence.
Improved classifier accuracy by using more of the class-specific structure in trained models.
problem Softmax ignores valuable information encoded in the full array of class response distributions.
method Developed a hybrid classifier (Softmax-Pooling Hybrid, SPH) that uses Softmax on high-scoring samples and a log-likelihood method on low-scoring samples. result Reduces test set error by 6% to 23% using the exact same trained model.