The study tightens risk bounds for mixtures of experts using local differential privacy.
problem Improving risk bounds for mixtures of experts.
method Imposing local differential privacy (LDP) on the gating mechanism of mixtures of experts.
result Theoretical bounds exhibit logarithmic dependence on the number of experts and tighter than existing bounds.
HS-MoE selects sparse experts using adaptive priors and data-adaptive gating.
problem Sparse expert selection in mixture-of-experts architectures.
method Combines horseshoe prior with input-dependent gating for data-adaptive sparsity.
result Data-adaptive sparsity in expert usage.
Bayesian models combine experts with a flexible gating mechanism for complex data.
problem Theoretical properties of Bayesian mixture-of-experts models with softmax gating remain unexplored.
method Investigated asymptotic behavior of posterior distribution for density estimation, parameter estimation, and model selection.
result Established posterior contraction rates for density estimation and parameter estimation, providing insights for practical model design.
Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
Proposes GPHMEs using Gaussian processes for hierarchical expert models.
problem Hierarchical mixtures of experts with complex gating functions.
method Gaussian process-gated hierarchical mixtures of experts (GPHMEs) with non-linear gating and expert functions.
result Outperforms tree-based HMEs and achieves good performance with reduced complexity.
VAEs struggle with surjective multimodal data, especially class labels describing images.
problem VAEs struggle to capture variability in surjective multimodal data.
method Theoretical and empirical demonstration of VAEs with a mixture of experts posterior.
result VAEs with a mixture of experts posterior can disregard variation in surjective multimodal data.
In this paper, we propose a novel mixture of expert architecture for learning polyhedral classifiers. We learn the parameters of the classifierusing an expectation maximization algorithm. Wederive the generalization bounds of the proposedapproach. Through an extensive simulation study, we show that the proposed method …
Paper analyzes approximation and learning of MoEs with (P)ReLU activation.
problem Scaling up deep learning models with MoEs and (P)ReLU activation.
method Approximation and learning-theoretic analysis of MoMLPs with (P)ReLU.
result MoMLPs can uniformly approximate Lipschitz functions with ε accuracy using O(ε−1) parameters. Proposes a partially linear structure to capture nonlinear relationships in mixture of experts models.
problem Suboptimal estimates due to linearity assumption in mixture of experts models.
method Introduces a partially linear structure that incorporates unspecified functions to capture nonlinear relationships.
result Establishes the identifiability of the proposed model under mild conditions and introduces a practical estimation algorithm.
Improved HMoE models using Laplace gating function enhance expert specialization and performance.
problem Improving performance of hierarchical mixture of experts models.
method Used Laplace gating function instead of Softmax in hierarchical mixture of experts models.
result Laplace gating function accelerates expert convergence and enhances specialization.
Higher granularity in MoE models boosts expressivity exponentially.
problem Expressivity of Mixture-of-Experts models with varying granularity.
method Comparing models with different numbers of active experts (granularity).
result Exponential separation in network expressivity based on granularity.
A new model predicts multivariate regression using similarities to data points.
problem Complex, high-dimensional input-output relationships.
method Bayesian mixture-of-experts with conditional Gaussian mixtures and variational Bayes.
result Outperforms competitors in high-dimensional settings.
Estimates parameters in a deviated Gaussian mixture model.
problem Testing goodness-of-fit between a known function and a mixture of experts.
method Constructs novel Voronoi-based loss functions to estimate parameters.
result Characterizes local convergence rates of parameter estimation more accurately.
Study improves stock price prediction using adaptive Mixture of Experts framework.
problem Tackles diverse volatility regimes in stock price prediction.
method Combines RNN for high-volatility stocks and linear regression for stable stocks with a gating mechanism.
result Achieves up to 33% improvement in MSE for volatile assets and 28% for stable assets.
Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.
problem Understanding dense MLP layers in LLMs.
method Theoretical connection between MoE models and SAE structure in activation space.
result MLP layers in LLMs can be well approximated by sparse Mixture of Experts.
Proposes φ-balancing for more balanced expert utilization in MoE models.
problem Balanced expert utilization in MoE models to avoid bias.
method Directly targets population-level balance by minimizing a convex potential function.
result Consistently outperforms prior methods in stability and effectiveness.
VMoER improves uncertainty quantification in MoE layers for scalable foundation models.
problem Uncertainty quantification in large-scale models like MoE layers.
method Structured Bayesian approach with amortized variational inference over routing logits and temperature parameter inference.
result Improves routing stability, reduces calibration error, and increases AUROC by 12%.
Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defines a soft decision tree where leaves correspond to experts and decision nodes correspond to gating …
A framework uses a mixture of predictors for semi-supervised inference.
problem Limited labeled data, abundant unlabeled data.
method Mixture of Experts (MOE) for semi-supervised inference.
result MOE-powered inference framework achieves smallest possible variance.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.
problem Efficient deployment of large language models constrained by static parameter footprint.
method Structured Trapezoidal ExpertNeuron Pruning (TENP) identifies and retains important experts and neurons.
result DeepSeek model achieves 10% better performance on code generation tasks with 40% expert sparsity.
Mixture of Experts (MoE) is a popular framework in the fields of statistics and machine learning for modeling heterogeneity in data for regression, classification and clustering. MoE for continuous data are usually based on the normal distribution. However, it is known that for data with asymmetric behavior, heavy tail…
ARGUE combines expert networks for anomaly detection.
problem Anomaly detection without labeled data.
method Gated mixture-of-experts architecture combining expert networks.
result Prior knowledge about normal data distribution is valuable.
This study analyzes communication constraints in MoE architectures using information theory.
problem Communication constraints in Mixture-of-Experts (MoE) architectures.
method Developed a rate-distortion characterization of finite-rate gating in MoE architectures using information theory.
result Yielded capacity-aware limits for communication-constrained MoE systems.
The paper analyzes EM for Mixtures of Experts and shows its equivalence to projected Mirror Descent.
problem Training Mixtures of Experts (MoE) models.
method Rigorously analyzes Expectation Maximization (EM) for MoE models using a Mirror Descent perspective.
result Derives new convergence results and identifies conditions for local linear convergence.
We provide a theoretical treatment of over-specified Gaussian mixtures of experts with covariate-free gating networks. We establish the convergence rates of the maximum likelihood estimation (MLE) for these models. Our proof technique is based on a novel notion of \emph{algebraic independence} of the expert functions. …
A method to improve LLMs by automating the construction of a mixture of expert prompts.
problem Limitation of single instruction prompts in covering complex problem spaces.
method Divide the problem space into sub-regions, each governed by a specialized expert with both an instruction and demos. A two-phase process constructs these experts.
result Achieves an average win rate of 81% across major benchmarks.
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
Crowdsourced training of large neural networks with decentralized Mixture-of-Experts.
problem Expensive training of large neural networks limits research contributions.
method Learning@home: decentralized Mixture-of-Experts for large, poorly connected participants.
result Performance and reliability of Learning@home surpass conventional distributed training.
Bayesian framework for analyzing heterogeneous covariance data with a novel MoE-Wishart model.
problem Analyzing complex multivariate systems with varying covariance structures.
method Comprehensive Bayesian framework using mixture-of-experts Wishart model with predictor-dependent mixture weights.
result Accurate subpopulation recovery and estimation in heterogeneous covariance scenarios.
MoEs can efficiently model complex tasks with low-dimensionality and sparsity.
problem Understanding the theoretical foundations of MoEs for complex tasks.
method Systematic study of MoEs with two structural priors: low-dimensionality and sparsity.
result MoEs can approximate functions on low-dimensional manifolds and exhibit exponential structured tasks.
DeepUnifiedMom uses deep learning to create better momentum portfolios.
problem Lack of unified momentum portfolios across different time frames.
method Multi-task learning with multi-gate mixture of experts.
result DeepUnifiedMom outperforms benchmark models in diverse asset classes.
New method estimates causal effects of multiple versions of treatment.
problem Ignoring multiple versions of treatment leads to biased causal effect estimates.
method Mixture-of-Experts framework for estimating version-specific causal effects.
result Effective method for estimating causal effects of latent versions.
Study compares MoE and RNN models for stock price prediction across volatility profiles.
problem Improving stock price prediction accuracy across different volatility levels.
method Dynamic Mixture of Experts model combining RNN and linear models, adjusting weights through a gating network.
result MoE model outperforms individual models in reducing prediction errors.
The mixture of experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from Sobolev space that is sufficiently differentiable and tha…
Develops a functional mix-of-experts model for multiclass classification.
problem Multiclass classification with univariate functional predictors.
method Functional mix-of-experts model with regularization and sparsity constraints.
result Regularized maximum likelihood estimation yields interpretable coefficient functions.
Study examines Lasso performance in high-dimensional MoE models.
problem Estimating MoE models in high-dimensional settings with Lasso.
method Investigates SGMoE models with Lasso regularization under mild assumptions.
result Provides non-asymptotic bounds for Lasso regularization parameter.
Training Gaussian process-based models typically involves an O(N3) computational bottleneck due to inverting the covariance matrix. Popular methods for overcoming this matrix inversion problem cannot adequately model all types of latent functions, and are often not parallelizable. However, judicious choice of model…
uMoE trains NNs with uncertain data by embedding uncertainty into training.
problem Managing aleatoric uncertainty in NN-based predictive models.
method Divide and Conquer strategy, Expert components, Gating Unit.
result uMoE outperforms baseline methods in uncertainty management.
N-BEATS-MOE improves time series forecasting by adapting to series characteristics.
problem Forecasting heterogeneous time series with varying characteristics.
method Mixture-of-Experts layer with dynamic block weighting.
result Consistent improvements across 12 benchmark datasets, especially for heterogeneous series.
Proposes a Varying-Coefficient MoE model for analyzing dynamic data.
problem Inadequate constant coefficients in MoE models for dynamic settings.
method Varying-Coefficient Mixture of Experts (VCMoE) model with varying coefficients in gating and expert models.
result Established identifiability and consistency of the VCMoE model.
We propose a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression. Our mixture-of-experts model is conceptually simple and hierarchically recombines computations for an overall approximation of a full Gaussian process. Closed-form and distributed computations allow for effici…
NAMEx merges experts using Nash bargaining for improved performance.
problem Sparse Mixture of Experts merging strategies lack a principled weighting mechanism.
method Reinterpreting expert merging through game theory, introducing Nash Merging and complex momentum.
result NAMEx consistently outperforms competing methods across various tasks and system sizes.
The paper establishes convergence rates for MoE models in classification problems.
problem Understanding the behavior of MoE models in classification settings.
method Established convergence rates for density and parameter estimation in softmax gating multinomial logistic MoE models.
result Parameter estimation rates are significantly improved with a novel modified softmax gating function.
Study analyzes convergence of parameter estimation in contaminated mixture of experts.
problem Challenges in learning from prompts in large-scale models.
method Convergence analysis, distinguishability condition, partial differential equations.
result Comprehensive convergence rates and minimax lower bounds for parameter estimation.
A new model improves the performance of knowledge distillation in GNNs.
problem Inconsistent performance of existing knowledge distillation techniques in GNNs.
method Proposes a new model, Routing-by-Memory (RbM), a form of Mixture-of-Experts (MoE), to address performance concerns.
result Demonstrates experimentally that RbM achieves considerably more consistent performance across multiple datasets.
Improved time series forecasting with expert loss integration.
problem Enhancing time series forecasting accuracy and efficiency.
method Adaptive Mixture-of-Experts framework with expert-specific loss integration and online learning.
result Significantly improved forecasting accuracy and computational efficiency.