HS-MoE selects sparse experts using adaptive priors and data-adaptive gating.
problem Sparse expert selection in mixture-of-experts architectures.
method Combines horseshoe prior with input-dependent gating for data-adaptive sparsity.
result Data-adaptive sparsity in expert usage.
The study tightens risk bounds for mixtures of experts using local differential privacy.
problem Improving risk bounds for mixtures of experts.
method Imposing local differential privacy (LDP) on the gating mechanism of mixtures of experts.
result Theoretical bounds exhibit logarithmic dependence on the number of experts and tighter than existing bounds.
Bayesian models combine experts with a flexible gating mechanism for complex data.
problem Theoretical properties of Bayesian mixture-of-experts models with softmax gating remain unexplored.
method Investigated asymptotic behavior of posterior distribution for density estimation, parameter estimation, and model selection.
result Established posterior contraction rates for density estimation and parameter estimation, providing insights for practical model design.
TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.
problem Efficient deployment of large language models constrained by static parameter footprint.
method Structured Trapezoidal ExpertNeuron Pruning (TENP) identifies and retains important experts and neurons.
result DeepSeek model achieves 10% better performance on code generation tasks with 40% expert sparsity.
Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
Improved HMoE models using Laplace gating function enhance expert specialization and performance.
problem Improving performance of hierarchical mixture of experts models.
method Used Laplace gating function instead of Softmax in hierarchical mixture of experts models.
result Laplace gating function accelerates expert convergence and enhances specialization.
Paper analyzes approximation and learning of MoEs with (P)ReLU activation.
problem Scaling up deep learning models with MoEs and (P)ReLU activation.
method Approximation and learning-theoretic analysis of MoMLPs with (P)ReLU.
result MoMLPs can uniformly approximate Lipschitz functions with ε accuracy using O(ε−1) parameters. A fast method combines deep mixtures of sparse GPs for flexible modeling.
problem Flexible modeling with changing output densities.
method Designing gating network with DNN for selecting sparse GPs, using CCR algorithm.
result The method outperforms competing methods in accuracy and uncertainty quantification.
This work improves Gaussian process inference using mixtures of experts and nested SMC samplers.
problem High computational and memory costs of Gaussian processes.
method Mixtures of Gaussian process experts with nested SMC samplers.
result Significantly improved inference compared to importance sampling.
Proposes a partially linear structure to capture nonlinear relationships in mixture of experts models.
problem Suboptimal estimates due to linearity assumption in mixture of experts models.
method Introduces a partially linear structure that incorporates unspecified functions to capture nonlinear relationships.
result Establishes the identifiability of the proposed model under mild conditions and introduces a practical estimation algorithm.
Estimates parameters in a deviated Gaussian mixture model.
problem Testing goodness-of-fit between a known function and a mixture of experts.
method Constructs novel Voronoi-based loss functions to estimate parameters.
result Characterizes local convergence rates of parameter estimation more accurately.
Study analyzes convergence of parameter estimation in contaminated mixture of experts.
problem Challenges in learning from prompts in large-scale models.
method Convergence analysis, distinguishability condition, partial differential equations.
result Comprehensive convergence rates and minimax lower bounds for parameter estimation.
Proposes GPHMEs using Gaussian processes for hierarchical expert models.
problem Hierarchical mixtures of experts with complex gating functions.
method Gaussian process-gated hierarchical mixtures of experts (GPHMEs) with non-linear gating and expert functions.
result Outperforms tree-based HMEs and achieves good performance with reduced complexity.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
ARGUE combines expert networks for anomaly detection.
problem Anomaly detection without labeled data.
method Gated mixture-of-experts architecture combining expert networks.
result Prior knowledge about normal data distribution is valuable.
Higher granularity in MoE models boosts expressivity exponentially.
problem Expressivity of Mixture-of-Experts models with varying granularity.
method Comparing models with different numbers of active experts (granularity).
result Exponential separation in network expressivity based on granularity.
Proposes φ-balancing for more balanced expert utilization in MoE models.
problem Balanced expert utilization in MoE models to avoid bias.
method Directly targets population-level balance by minimizing a convex potential function.
result Consistently outperforms prior methods in stability and effectiveness.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
Improved time series forecasting with expert loss integration.
problem Enhancing time series forecasting accuracy and efficiency.
method Adaptive Mixture-of-Experts framework with expert-specific loss integration and online learning.
result Significantly improved forecasting accuracy and computational efficiency.
NAMEx merges experts using Nash bargaining for improved performance.
problem Sparse Mixture of Experts merging strategies lack a principled weighting mechanism.
method Reinterpreting expert merging through game theory, introducing Nash Merging and complex momentum.
result NAMEx consistently outperforms competing methods across various tasks and system sizes.
A method to improve LLMs by automating the construction of a mixture of expert prompts.
problem Limitation of single instruction prompts in covering complex problem spaces.
method Divide the problem space into sub-regions, each governed by a specialized expert with both an instruction and demos. A two-phase process constructs these experts.
result Achieves an average win rate of 81% across major benchmarks.
The paper analyzes EM for Mixtures of Experts and shows its equivalence to projected Mirror Descent.
problem Training Mixtures of Experts (MoE) models.
method Rigorously analyzes Expectation Maximization (EM) for MoE models using a Mirror Descent perspective.
result Derives new convergence results and identifies conditions for local linear convergence.
A new model predicts multivariate regression using similarities to data points.
problem Complex, high-dimensional input-output relationships.
method Bayesian mixture-of-experts with conditional Gaussian mixtures and variational Bayes.
result Outperforms competitors in high-dimensional settings.
The paper develops methods to create reliable prediction sets for complex mixture models in high-dimensional data.
problem Building accurate prediction sets for high-dimensional mixture models with feature-dependent weights.
method The authors introduce a debiasing procedure and a novel interval combination strategy to construct valid prediction sets.
result The proposed method provides reliable coverage guarantees for prediction sets in high-dimensional mixture models.
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
This paper explores how MoE layers improve deep learning performance.
problem Understanding the Mixture-of-Experts (MoE) layer in deep learning.
method Formal study of MoE layer's effectiveness and mechanism.
result MoE layer improves performance by leveraging cluster structure and non-linearity.
Paper proposes a novel method to accurately determine the number of experts in Gaussian-gated Gaussian MoE models.
problem Challenges in model selection for MoE models, especially with covariates.
method Introduces a novel extension using dendrograms of mixing measures to estimate the true number of mixture components.
result Achieves optimal convergence rates for parameter estimation and accurately approximates the regression function.
VMoER improves uncertainty quantification in MoE layers for scalable foundation models.
problem Uncertainty quantification in large-scale models like MoE layers.
method Structured Bayesian approach with amortized variational inference over routing logits and temperature parameter inference.
result Improves routing stability, reduces calibration error, and increases AUROC by 12%.
Mixtures of experts probabilistically divide the input space into regions, where the assumptions of each expert, or conditional model, need only hold locally. Combined with Gaussian process (GP) experts, this results in a powerful and highly flexible model. We focus on alternative mixtures of GP experts, which model th…
Sepsis is the leading cause of mortality in the ICU. It is challenging to manage because individual patients respond differently to treatment. Thus, tailoring treatment to the individual patient is essential for the best outcomes. In this paper, we take steps toward this goal by applying a mixture-of-experts framework …
The paper proposes methods to identify and sample from mixtures of Mallows models for top-k rankings.
problem Identifying and sampling from mixtures of Mallows models for top-k rankings in a heterogeneous population.
method Efficient sampling algorithms and identifiability proofs for both components of the mixture.
result The identifiability and learnability of the Mallows components' parameters in the mixture.
Develops a functional mix-of-experts model for multiclass classification.
problem Multiclass classification with univariate functional predictors.
method Functional mix-of-experts model with regularization and sparsity constraints.
result Regularized maximum likelihood estimation yields interpretable coefficient functions.
fMRI semantic category understanding using linguistic encoding models attempt to learn a forward mapping that relates stimuli to the corresponding brain activation. Classical encoding models use linear multi-variate methods to predict the brain activation (all voxels) given the stimulus. However, these methods essentia…
Improved Gaussian process experts model for complex data.
problem Limitations of standard Gaussian processes: scalability and predictive performance.
method Proposes a new mixture model of Gaussian process experts based on kernel stick-breaking processes.
result Improved predictive performance compared to existing models.
Computations for the softmax function are significantly expensive when the number of output classes is large. In this paper, we present a novel softmax inference speedup method, Doubly Sparse Softmax (DS-Softmax), that leverages sparse mixture of sparse experts to efficiently retrieve top-k classes. Different from most…
VAEs struggle with surjective multimodal data, especially class labels describing images.
problem VAEs struggle to capture variability in surjective multimodal data.
method Theoretical and empirical demonstration of VAEs with a mixture of experts posterior.
result VAEs with a mixture of experts posterior can disregard variation in surjective multimodal data.
A framework uses a mixture of predictors for semi-supervised inference.
problem Limited labeled data, abundant unlabeled data.
method Mixture of Experts (MOE) for semi-supervised inference.
result MOE-powered inference framework achieves smallest possible variance.
New insights into the top-K sparse softmax gating function for deep learning.
problem Understanding the theoretical effects of the top-K sparse softmax gating function on density and parameter estimations.
method Using a Gaussian mixture of experts, novel loss functions, and theoretical analysis.
result The convergence rates of density and parameter estimations are parametric under certain conditions, but slow under over-specified models.
Mixture of Experts (MoE) is a popular framework in the fields of statistics and machine learning for modeling heterogeneity in data for regression, classification and clustering. MoE for continuous data are usually based on the normal distribution. However, it is known that for data with asymmetric behavior, heavy tail…
This study analyzes communication constraints in MoE architectures using information theory.
problem Communication constraints in Mixture-of-Experts (MoE) architectures.
method Developed a rate-distortion characterization of finite-rate gating in MoE architectures using information theory.
result Yielded capacity-aware limits for communication-constrained MoE systems.
In this paper, we propose a novel mixture of expert architecture for learning polyhedral classifiers. We learn the parameters of the classifierusing an expectation maximization algorithm. Wederive the generalization bounds of the proposedapproach. Through an extensive simulation study, we show that the proposed method …
Study improves stock price prediction using adaptive Mixture of Experts framework.
problem Tackles diverse volatility regimes in stock price prediction.
method Combines RNN for high-volatility stocks and linear regression for stable stocks with a gating mechanism.
result Achieves up to 33% improvement in MSE for volatile assets and 28% for stable assets.
Gaussian Processes (GPs) are powerful non-parametric Bayesian regression models that allow exact posterior inference, but exhibit high computational and memory costs. In order to improve scalability of GPs, approximate posterior inference is frequently employed, where a prominent class of approximation techniques is ba…
MoEs can efficiently model complex tasks with low-dimensionality and sparsity.
problem Understanding the theoretical foundations of MoEs for complex tasks.
method Systematic study of MoEs with two structural priors: low-dimensionality and sparsity.
result MoEs can approximate functions on low-dimensional manifolds and exhibit exponential structured tasks.
Paper proposes interpretable RL policies from a mixture of experts.
problem Making RL policies transparent and understandable in real-world applications.
method Policy iteration scheme with interpretable experts and prototypical states.
result Proposed algorithm learns policies comparable to neural networks but more interpretable.
LMoE uses LLMs to improve stock trading by selecting experts based on textual and price data.
problem Traditional neural network-based router selection in MoE models is suboptimal and ignores textual data.
method Proposes LLMoE, using LLMs as routers to select experts based on historical price data and stock news.
result LLoM outperforms state-of-the-art MoE models and other deep neural network approaches.
The paper analyzes convergence rates of softmax gating in MoE models.
problem The effectiveness and scalability of machine learning models using MoE.
method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.
result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.
Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defines a soft decision tree where leaves correspond to experts and decision nodes correspond to gating …