On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

14 results for “Softmax-Gated”

Demystifying Softmax Gating Function in Gaussian Mixture of Expertsstat.ML

Solves challenges in estimating parameters of softmax gating Gaussian mixture models.

problem Identifiability issues and complex interactions in Gaussian mixture of experts.

method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.

result Connects convergence rate of MLE to a solvability problem of polynomial equations.

softmax gating Gaussian mixture parameter estimation MLE identifiability

2023-05-05abs ↗pdf ↗

Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertsstat.ML

New insights into the top-K sparse softmax gating function for deep learning.

problem Understanding the theoretical effects of the top-K sparse softmax gating function on density and parameter estimations.

method Using a Gaussian mixture of experts, novel loss functions, and theoretical analysis.

result The convergence rates of density and parameter estimations are parametric under certain conditions, but slow under over-specified models.

top-k sparse softmax gating function density estimation parameter estimation Voronoi cells

2023-09-25abs ↗pdf ↗

Convergence Rates for Softmax Gating Mixture of Expertsstat.ML

The paper analyzes convergence rates of softmax gating in MoE models.

problem The effectiveness and scalability of machine learning models using MoE.

method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.

result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.

Mixture of Experts Softmax Gating Parameter Estimation Convergence Analysis Machine Learning

2025-03-05abs ↗pdf ↗

A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertsstat.ML

The paper establishes convergence rates for MoE models in classification problems.

problem Understanding the behavior of MoE models in classification settings.

method Established convergence rates for density and parameter estimation in softmax gating multinomial logistic MoE models.

result Parameter estimation rates are significantly improved with a novel modified softmax gating function.

Mixture-of-experts MoE softmax gating multinomial logistic classification

2023-10-22abs ↗pdf ↗

On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionsstat.ML

Improved HMoE models using Laplace gating function enhance expert specialization and performance.

problem Improving performance of hierarchical mixture of experts models.

method Used Laplace gating function instead of Softmax in hierarchical mixture of experts models.

result Laplace gating function accelerates expert convergence and enhances specialization.

Hierarchical Mixture of Experts Laplace Gating Softmax Gating Expert Specialization Performance Improvement

2024-10-03abs ↗pdf ↗

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertsstat.ML

Sigmoid gating is more sample efficient than softmax in mixture of experts.

problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.

method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.

result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.

mixture of experts sigmoid gating softmax gating sample efficiency representation collapse

2024-05-22abs ↗pdf ↗

Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of expertsmath.ST

Study examines Lasso performance in high-dimensional MoE models.

problem Estimating MoE models in high-dimensional settings with Lasso.

method Investigates SGMoE models with Lasso regularization under mild assumptions.

result Provides non-asymptotic bounds for Lasso regularization parameter.

Mixture of experts Lasso regularization High-dimensional estimation Softmax gating Gaussian experts

2020-09-22abs ↗pdf ↗

Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Modelsstat.ML

We improve MoE models for classification with rigorous guarantees and practical methods.

problem Limited guarantees for stable maximum-likelihood training and model selection in softmax-gated MoE models.

method Derived a batch MM algorithm with closed-form updates, proved finite-sample rates, and developed a dendrogram selector.

result Achieved near-parametric optimal rates for parameter recovery and improved accuracy over baselines.

Mixture-of-Experts Softmax-Gated Multinomial-Logistic Model Selection Stable Optimization

2026-02-08abs ↗pdf ↗

Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepsstat.ML

Unified framework for SGMoE resolves estimation and selection issues.

problem Non-identifiability, coupled differential relations, and tight coupling in softmax-Gated models.

method Unified statistical framework with Voronoi-type loss functions and dendrograms of mixing measures.

result Consistent selection of the number of experts without model sweeps, optimal parameter rates under overfitting.

Gaussian mixture of experts softmax gating non-identifiability dendrograms consistent selection

2025-10-14abs ↗pdf ↗

On Bayesian Softmax-Gated Mixture-of-Experts Modelsstat.ML

Bayesian models combine experts with a flexible gating mechanism for complex data.

problem Theoretical properties of Bayesian mixture-of-experts models with softmax gating remain unexplored.

method Investigated asymptotic behavior of posterior distribution for density estimation, parameter estimation, and model selection.

result Established posterior contraction rates for density estimation and parameter estimation, providing insights for practical model design.

Bayesian Mixture-of-Experts Softmax Posterior Contraction Density Estimation

2026-04-22abs ↗pdf ↗

On Least Square Estimation in Softmax Gating Mixture of Expertsstat.ML

Investigates least squares estimation in deterministic MoE models.

problem Complexity and difficulty in analyzing MoE models.

method Least squares estimation under deterministic MoE models.

result Rates of estimating strongly identifiable experts are faster than polynomial experts.

MoE models least squares estimation strong identifiability sigmoid tanh

2024-02-05abs ↗pdf ↗

Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functionstat.ML

Improved logistic MoE with sigmoid gate shows better sample efficiency.

problem Improving sample efficiency in logistic MoE models.

method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.

result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.

logistic MoE sigmoid gate sample efficiency temperature parameter Euclidean score

2026-02-01abs ↗pdf ↗

Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertsstat.ML

New algorithm improves on EM for streaming data, outperforming existing methods.

problem Processing high-volume, streaming data efficiently.

method Incremental stochastic Majorization-Minimization (MM) algorithm.

result The algorithm converges to a stationary point with vanishing gradient.

streaming data incremental algorithms majorization-minimization EM algorithm stochastic optimization

2026-01-27abs ↗pdf ↗

Gradient-free variational learning with conditional mixture networkscs.LG

Gradient-free method improves predictive accuracy for probabilistic models.

problem Balancing computational efficiency and robust predictive performance in deep learning.

method CAVI-CMN, a gradient-free variational method for conditional mixture networks.

result CAVI-CMN achieves competitive and often superior predictive accuracy compared to MLE with backpropagation.

Bayesian learning variational inference conditional mixture networks gradient-free optimization deep learning

2024-08-29abs ↗pdf ↗