A new framework for robust policy learning in MDPs with linear mixture dynamics.
problem Off-dynamics challenge in real-world decision-making problems.
method Linear mixture DRMDP framework, meta algorithm for robust policy learning.
result The new framework provides a more refined representation of uncertainties.
Study on recovering supports of multiple sparse vectors from mixed linear measurements.
problem Recovering supports of multiple sparse vectors from a mixture of linear measurements.
method Developed algorithms to identify the support of all component vectors using polynomial and quasi-polynomial number of measurements.
result Polynomial and quasi-polynomial number of measurements sufficient for recovering the supports of all component vectors.
New approach learns mixtures of linear dynamical systems without separation conditions.
problem Learning mixtures of linear dynamical systems with better fit or understanding.
method Tensor decompositions to learn mixtures of linear dynamical systems.
result Algorithm succeeds without strong separation conditions and can compete with Bayes optimal clustering.
Proposes a partially linear structure to capture nonlinear relationships in mixture of experts models.
problem Suboptimal estimates due to linearity assumption in mixture of experts models.
method Introduces a partially linear structure that incorporates unspecified functions to capture nonlinear relationships.
result Establishes the identifiability of the proposed model under mild conditions and introduces a practical estimation algorithm.
Robust learning mixtures of linear regressions improve robustness.
problem Improving robustness in learning mixtures of linear regressions.
method Connecting mixtures of linear regressions and mixtures of Gaussians with thresholding for a quasi-polynomial time algorithm.
result The algorithm has significantly better robustness than previous results.
Estimates MLDS using tensor decomposition, improving upon existing methods.
problem Learning mixtures of linear dynamical systems from input-output data.
method Proposes a moment-based estimator using tensor decomposition.
result Improves sample complexity bounds for estimating MLDS.
Efficient algorithms for sparse parameter recovery in mixture models.
problem Support recovery of high-dimensional sparse latent vectors in mixture models.
method Efficient algorithms with logarithmic sample complexity dependence on dimensionality.
result First guarantees on support recovery for various mixture models.
Linear mixture models have proven very useful in a plethora of applications, e.g., topic modeling, clustering, and source separation. As a critical aspect of the linear mixture models, identifiability of the model parameters is well-studied, under frameworks such as independent component analysis and constrained matrix…
The paper develops methods to create reliable prediction sets for complex mixture models in high-dimensional data.
problem Building accurate prediction sets for high-dimensional mixture models with feature-dependent weights.
method The authors introduce a debiasing procedure and a novel interval combination strategy to construct valid prediction sets.
result The proposed method provides reliable coverage guarantees for prediction sets in high-dimensional mixture models.
A Gaussian mixture model improves generalization for long-tailed data.
problem Optimizing generalization for rare data in long-tailed distributions.
method Suggested Gaussian mixture model and comparison of linear vs. nonlinear classifiers.
result Nonlinear classifiers outperform linear ones for long-tailed data.
New methods combine model predictions to avoid linear mixtures' limitations.
problem Combining predictions from different models to avoid linear mixtures' limitations.
method Log-linear pooling (locking) and quantum superposition (quacking) to optimise model weights.
result Demonstrated locking method with illustrative example and practical application.
Method estimates mixture components without discretizing parameters.
problem Learning from mixtures of continuous features with noise.
method Off-the-grid optimization method for continuous parameter space.
result Prediction error bound similar to Lasso predictor.
The study characterizes learning Gaussian mixtures using GLMs in high dimensions.
problem Learning Gaussian mixtures with generalised linear models in high-dimensional settings.
method Empirical risk minimization with convex loss and regularisation.
result Exact asymptotics of the ERM estimator for Gaussian mixtures in high dimensions.
Transformers can learn mixture of linear models efficiently.
problem Existence and generalization of in-context learning for mixture models.
method Theoretical analysis and gradient flow optimization.
result Transformers achieve a prediction error of O ( d / n ) \mathcal{O}(\sqrt{d/n}) O ( d / n ) with high probability. The study uncovers universality laws for Gaussian mixtures in generalized linear models.
problem Understanding the asymptotic behavior of estimators in Gaussian mixture models.
method Investigates the asymptotic joint statistics of generalized linear estimators from empirical risk minimization and Gibbs sampling.
result Characterizes conditions under which the joint statistics depend only on means and covariances of class conditional features.
Deep learning is a hierarchical inference method formed by subsequent multiple layers of learning able to more efficiently describe complex relationships. In this work, Deep Gaussian Mixture Models are introduced and discussed. A Deep Gaussian Mixture model (DGMM) is a network of multiple layers of latent variables, wh…
Tensor decomposition recovers Gaussian mixtures from moments.
problem Recovering Gaussian mixture models from datasets.
method Symmetric tensor decomposition of moment tensors built from empirical moments.
result Identifiable tensors with interpolation degree less than half their order.
Optimizes mixture models without parametrizing distributions using tensor decomposition.
problem Estimating conditionally-independent mixture models in high dimensions.
method Alternating least squares optimization scheme for tensor decomposition.
result Competitive performance and applicability to various models and applications.
The paper uses Gaussian mixture models for Bayesian networks and proposes an optimization algorithm.
problem Modeling nodes in Bayesian networks with complex distributions.
method Gaussian mixture models combined with double iteration algorithm.
result The double iteration algorithm optimizes Gaussian mixture models effectively.
New bounds on sample size for identifying mixture models with grouped samples.
problem Identifying mixture models with minimal sample size.
method Generalized identifiability bounds for mixture models with grouped samples.
result Identifiability with ( 2 m − 1 ) / ( k − 1 ) (2m-1)/(k-1) ( 2 m − 1 ) / ( k − 1 ) samples per group, with no improvement possible. Two EM algorithms estimate prior distributions in mixture of linear regressions.
problem Estimating prior distributions in mixture of linear regressions.
method Two EM algorithms: one for continuous priors, one for discrete priors.
result Both algorithms accurately estimate prior distributions and the number of clusters.
Proposes GPHMEs using Gaussian processes for hierarchical expert models.
problem Hierarchical mixtures of experts with complex gating functions.
method Gaussian process-gated hierarchical mixtures of experts (GPHMEs) with non-linear gating and expert functions.
result Outperforms tree-based HMEs and achieves good performance with reduced complexity.
New analysis proves sketching operators' RIP guarantees for mixture models without importance sampling.
problem Proving sketching operators' Restricted Isometry Property (RIP) for mixture models without assuming importance sampling.
method Proposed alternative analysis based on new deterministic bounds and concentration inequalities.
result Theoretical guarantees for sketching operators without importance sampling.
Paper uses Gaussian mixture models and Wasserstein distance for schema matching.
problem Schema matching between different datasets.
method Gaussian mixture models and Wasserstein distance for comparison.
result Derives an approximation for Wasserstein distance between Gaussian mixture models.
The Expectation-Maximization algorithm is perhaps the most broadly used algorithm for inference of latent variable problems. A theoretical understanding of its performance, however, largely remains lacking. Recent results established that EM enjoys global convergence for Gaussian Mixture Models. For Mixed Linear Regres…
New algorithm learns optimal path in reinforcement learning with linear approximations.
problem Optimal path learning in reinforcement learning with linear approximations.
method Proposes novel algorithm with Hoeffding-type and Bernstein-type confidence sets.
result Achieves near-optimal regret guarantee for linear mixture SSP.
A mixture of factor analyzers is a semi-parametric density estimator that generalizes the well-known mixtures of Gaussians model by allowing each Gaussian in the mixture to be represented in a different lower-dimensional manifold. This paper presents a robust and parsimonious model selection algorithm for training a mi…
Gradient method converges locally linearly for overparameterized Gaussian mixtures.
problem Learning Gaussian mixtures under overparameterization.
method Gradient-based method alternating short descent steps and long Polyak steps.
result Gradient method converges locally linearly to minimizers.
New findings show Gaussian universality breaks down in high-dimensional linear factor mixtures.
problem The limitations of Gaussian universality in high-dimensional classification.
method Characterization of empirical risk minimization for classification under linear factor mixture models.
result Gaussian universality breaks down under high-dimensional linear factor mixtures.
We propose Dirichlet Process mixtures of Generalized Linear Models (DP-GLM), a new method of nonparametric regression that accommodates continuous and categorical inputs, and responses that can be modeled by a generalized linear model. We prove conditions for the asymptotic unbiasedness of the DP-GLM regression mean fu…
The paper analyzes EM for Mixtures of Experts and shows its equivalence to projected Mirror Descent.
problem Training Mixtures of Experts (MoE) models.
method Rigorously analyzes Expectation Maximization (EM) for MoE models using a Mirror Descent perspective.
result Derives new convergence results and identifies conditions for local linear convergence.
Gradient EM converges exponentially to optimal solution in agnostic mixtures.
problem Fitting k k k parametric functions to given data points without a generative model. method Gradient EM algorithm for agnostic mixtures of arbitrary parametric functions.
result Gradient EM converges exponentially to population loss minimizers with high probability.
Algorithm learns multiple LDS models from unlabeled short trajectories.
problem Learning from unlabeled short sample trajectories of multiple LDS models.
method Two-stage meta-algorithm with end-to-end performance guarantees.
result Efficiently recovers each ground-truth LDS model up to error i l d e O ( d / T ) ilde{O}(\sqrt{d/T}) i l d e O ( d / T ) . The paper tackles learning mixtures of two multinomial logits, showing identifiability and presenting an algorithm.
problem Learning an arbitrary mixture of two multinomial logits.
method Reduction to solving a system of univariate quartic equations, followed by an algorithm using polynomial and linear samples.
result Identifiability of the mixture models may only fail on an algebraic variety of negligible measure.
Study compares MoE and RNN models for stock price prediction across volatility profiles.
problem Improving stock price prediction accuracy across different volatility levels.
method Dynamic Mixture of Experts model combining RNN and linear models, adjusting weights through a gating network.
result MoE model outperforms individual models in reducing prediction errors.
Discriminative latent-variable models are typically learned using EM or gradient-based optimization, which suffer from local optima. In this paper, we develop a new computationally efficient and provably consistent estimator for a mixture of linear regressions, a simple instance of a discriminative latent-variable mode…
Mix-IRLS solves imbalanced mixed linear regression problems efficiently.
problem Imbalanced mixed linear regression problems.
method Sequential robust regression approach.
result Mix-IRLS outperforms other methods on imbalanced mixtures and real-world datasets.
New method for summarizing Bayesian mixture models using sliced Wasserstein distances.
problem Estimating the mixing measure in nonparametric Bayesian mixture models.
method Decision-theoretic approach using sliced Wasserstein distances for Gaussian mixtures.
result Effective estimation of the mixing measure and mixture density.
Lower bounds show learning mixtures of linear classifiers is nearly impossible.
problem Learning mixtures of linear classifiers under Gaussian covariates.
method Statistical Query (SQ) lower bounds and new spherical designs.
result Complexity of any SQ algorithm is \( n^{\mathrm{poly}(1/Δ) \log(r)} \), where Δ is the pairwise \(\ell_2\)-separation.
New algorithm reduces regret in linear mixture SSPs without cost bounds.
problem Learning optimal paths in stochastic environments with cost constraints.
method Extended value iteration with variance-aware confidence set.
result Achieves nearly minimax optimal regret bound of O ( d B ∗ K ) O(dB_*\sqrt{K}) O ( d B ∗ K ) . Wasserstein framework solves mixed linear regression problems.
problem Mixed linear regression with multi-modal distributions.
method Wasserstein distance minimization for nonconvex-concave minimax optimization.
result WMLR achieves global convergence and generalization guarantees for two linear models.
A Bayesian filtering algorithm is developed for a class of state-space systems that can be modelled via Gaussian mixtures. In general, the exact solution to this filtering problem involves an exponential growth in the number of mixture terms and this is handled here by utilising a Gaussian mixture reduction step after …
Paper tackles MLR prediction error without assuming realizable models.
problem Prediction error in mixture of linear regressions without realizable assumptions.
method Developed algorithms for list-decoding MLR predictions and minimized empirical risk.
result Alternating minimization algorithm finds best fit lines in non-realizable settings.
This paper studies generalization in machine learning with mixture data.
problem Generalization performance and statistical rates in heterogeneous data.
method Characterization of heterogeneity via pairwise total variation distance, analysis of Rademacher and Gaussian complexities.
result The requirement on heterogeneity increases as function classes get more complex.
We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable …
This paper is a step-by-step tutorial for fitting a mixture distribution to data. It merely assumes the reader has the background of calculus and linear algebra. Other required background is briefly reviewed before explaining the main algorithm. In explaining the main algorithm, first, fitting a mixture of two distribu…
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Mixture models with Gamma and or inverse-Gamma distributed mixture components are useful for medical image tissue segmentation or as post-hoc models for regression coefficients obtained from linear regression within a Generalised Linear Modeling framework (GLM), used in this case to separate stochastic (Gaussian) noise…