arXiv research

TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.

A method to select important experts for Gaussian processes to balance computational efficiency and uncertainty quantification.

Expert augmentation improves hybrid model generalization.

Bayesian models combine experts with a flexible gating mechanism for complex data.

New method calibrates Gaussian product experts for better predictions.

HS-MoE selects sparse experts using adaptive priors and data-adaptive gating.

Improved HMoE models using Laplace gating function enhance expert specialization and performance.

In this article, we aim at improving the prediction of expert aggregation by using the underlying properties of the models that provide expert predictions. We restrict ourselves to the case where expert predictions come from Kalman recursions, fitting state-space models. By using exponential weights, we construct diffe…

Improved time series forecasting with expert loss integration.

DeepSeekMoE improves language model efficiency with shared experts and normalized gating.

In order to improve forecasts, a decisionmaker often combines probabilities given by various sources, such as human experts and machine learning classifiers. When few training data are available, aggregation can be improved by incorporating prior knowledge about the event being forecasted and about salient properties o…

We present a new approach for learning compact and intuitive distributed representations with binary encoding. Rather than summing up expert votes as in products of experts, we employ for each variable the opinion of the most reliable expert. Data points are hence explained through a partitioning of the variables into …

The paper introduces a method to incorporate expert opinion on observable quantities into statistical models.

fMRI semantic category understanding using linguistic encoding models attempt to learn a forward mapping that relates stimuli to the corresponding brain activation. Classical encoding models use linear multi-variate methods to predict the brain activation (all voxels) given the stimulus. However, these methods essentia…

The paper combines Bitcoin price models with expert corrections for better predictions.

Unified approach to aggregating models and preferences.

Investigates least squares estimation in deterministic MoE models.

Improves accuracy and fairness in prediction systems with multiple domain experts.

A method merges two pretrained diffusion experts to improve image quality and likelihood.

New method aggregates Gaussian experts by detecting conditional independence violations.

EBBS integrates expert assessments into MIO best-subsets problem.

NAMEx merges experts using Nash bargaining for improved performance.

In this paper we introduce a model of lifelong learning, based on a Network of Experts. New tasks / experts are learned and added to the model sequentially, building on what was learned before. To ensure scalability of this process,data from previous tasks cannot be stored and hence is not available when learning a new…

Improved regret bounds for bandits with expert advice.

Higher granularity in MoE models boosts expressivity exponentially.

Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by e…

CoCoAFusE fuses expert predictions to model complex patterns with interpretability and uncertainty.

Framework allows systems to defer difficult decisions to unknown experts.

Proposes $φ$-balancing for more balanced expert utilization in MoE models.

Proposes unbiased estimators for training mixture of experts models.

This work improves Gaussian process inference using mixtures of experts and nested SMC samplers.

In regression tasks the distribution of the data is often too complex to be fitted by a single model. In contrast, partition-based models are developed where data is divided and fitted by local models. These models partition the input space and do not leverage the input-output dependency of multimodal-distributed data,…

Paper connects MoE and self-attention, proposing active-attention.

The paper analyzes convergence rates of softmax gating in MoE models.

High-dimensional prediction is a challenging problem setting for traditional statistical models. Although regularization improves model performance in high dimensions, it does not sufficiently leverage knowledge on feature importances held by domain experts. As an alternative to standard regularization techniques, we p…

A fast method combines deep mixtures of sparse GPs for flexible modeling.

Sigmoid gating is more sample efficient than softmax in mixture of experts.

Develops a functional mix-of-experts model for multiclass classification.

Existing imitation learning approaches often require that the complete demonstration data, including sequences of actions and states, are available. In this paper, we consider a more realistic and difficult scenario where a reinforcement learning agent only has access to the state sequences of an expert, while the expe…

Learning from demonstration (LfD) is the process of building behavioral models of a task from demonstrations provided by an expert. These models can be used e.g. for system control by generalizing the expert demonstrations to previously unencountered situations. Most LfD methods, however, make strong assumptions about …

Multi-expert L2D underfits more severely, requiring new methods.

New findings explain why online methods outperform offline methods in noisy expert feedback settings.

MM-DREX adapts LLM experts for financial trading via dynamic routing.

Proposes GPHMEs using Gaussian processes for hierarchical expert models.

Improved transfer learning with expert models, reducing compute and speeding up performance.

VMoER improves uncertainty quantification in MoE layers for scalable foundation models.

ARGUE combines expert networks for anomaly detection.

Mixtures of experts probabilistically divide the input space into regions, where the assumptions of each expert, or conditional model, need only hold locally. Combined with Gaussian process (GP) experts, this results in a powerful and highly flexible model. We focus on alternative mixtures of GP experts, which model th…