This paper compares EM and GD in two-component mixture models, finding EM escapes bad local optima more reliably.
problem Understanding the convergence of EM and GD in mixture models, especially in regions where one component is missing.
method Analyzing regions called one-cluster regions in two-component mixture models of Gaussians and Bernoullis, comparing the propensity of EM and GD to converge to these regions.
result EM escapes one-cluster regions exponentially fast, while GD escapes them linearly fast, indicating EM is less likely to converge to bad local optima.
When estimating finite mixture models, it is common to make assumptions on the mixture components, such as parametric assumptions. In this work, we make no distributional assumptions on the mixture components and instead assume that observations from the mixture model are grouped, such that observations in the same gro…
Study uniform rates for estimating Gaussian mixtures without separation assumption.
problem Estimating parameters in two-component Gaussian mixtures without separation.
method Uniform convergence rates derived using minimax lower bounds and careful analysis of polynomial equalities.
result Phase transition in optimal estimation rate based on mixture balance.
New method estimates mixture model components efficiently.
problem Estimating the number of components in finite mixture models.
method Group-Sort-Fuse (GSF) procedure for simultaneous estimation of order and mixing measure.
result GSF achieves consistent estimation of true mixture order and n−1/2 convergence rate. New model separates object attributes for better perceptual grouping.
problem Perceptual grouping of complex visual scenes.
method Spatial mixture models with learnable priors.
result Outperforms state-of-the-art methods in perceptual grouping.
The two most extended density-based approaches to clustering are surely mixture model clustering and modal clustering. In the mixture model approach, the density is represented as a mixture and clusters are associated to the different mixture components. In modal clustering, clusters are understood as regions of high d…
Study reveals structure of local minima in GMMs, identifying key cluster centers.
problem Identifying optimal cluster centers in non-convex GMM landscapes.
method Analyzing the negative log-likelihood function of GMMs in the population limit.
result Local minima share a common structure that partially identifies true cluster centers.
Improves scalability and efficiency of mixture models in black-box variational inference.
problem Scaling mixture models in black-box variational inference leads to high parameter and time costs.
method Introduces MISVAE for amortized mixture parameter space and new ELBO estimators.
result Achieves superior estimation performance with fewer parameters and shorter inference time.
Bayesian approach learns nonparametric mixture components from heterogeneous data.
problem Realistic modeling of heterogeneous data populations with nonparametric mixture components.
method Bayesian nonparametric modeling using Dirichlet process mixture priors.
result Posterior contraction rates for component densities are nearly polynomial, improving over deconvolution methods.
Study learns mixtures of smooth product distributions from samples.
problem Learning mixtures of non-parametric product distributions.
method Two-stage approach using identifiability properties of tensor decomposition and signal processing techniques.
result Recovery of component distributions under a smoothness condition.
We consider unsupervised estimation of mixtures of discrete graphical models, where the class variable corresponding to the mixture components is hidden and each mixture component over the observed variables can have a potentially different Markov graph structure and parameters. We propose a novel approach for estimati…
Researchers show mixtures of ranking models are generally identifiable.
problem Understanding when and how parameters of mixtures of ranking models can be uniquely determined.
method Algebraic geometry framework applied to verify the number of solutions in polynomial systems.
result Popular mixtures of ranking models with two components are generically identifiable.
New bounds on sample size for identifying mixture models with grouped samples.
problem Identifying mixture models with minimal sample size.
method Generalized identifiability bounds for mixture models with grouped samples.
result Identifiability with (2m−1)/(k−1) samples per group, with no improvement possible. Detecting and recovering labels in binomial logistic mixtures is challenging due to an information gap.
problem Detecting and recovering labels in binomial logistic mixtures
method Propose two feasibility-aware inference procedures
result Avoid misleading component selections and improve label probability calibration
Paper tackles learning mixture of RUMs from partial data.
problem Learning a mixture of Random Utility Models (RUMs) from pairwise comparisons.
method PCA-based spectral clustering to reduce mixture to single component.
result Algorithm correctly clusters data from a mixture of RUMs with high probability.
Algorithm distinguishes Gaussian mixtures from pure Gaussians in quasi-polynomial time.
problem Distinguishing mixtures of Gaussian components from pure Gaussians, especially when components are well-separated.
method Sum-of-Squares method, quasi-polynomial time algorithm, bipartitioning sample to separate components.
result Algorithm can reliably distinguish between mixtures and pure Gaussians in quasi-polynomial time.
We present the multidimensional membership mixture (M3) models where every dimension of the membership represents an independent mixture model and each data point is generated from the selected mixture components jointly. This is helpful when the data has a certain shared structure. For example, three unique means and …
Enhances mixture models with classifier-defined weights.
problem Density evaluation and sampling in mixture models.
method Introduces Classifier Weighted Mixtures (CWM) with functional weights.
result Improves expressivity in variational estimation without increasing complexity.
Clustering of data sets is a standard problem in many areas of science and engineering. The method of spectral clustering is based on embedding the data set using a kernel function, and using the top eigenvectors of the normalized Laplacian to recover the connected components. We study the performance of spectral clust…
In mixture model-based clustering applications, it is common to fit several models from a family and report clustering results from only the `best' one. In such circumstances, selection of this best model is achieved using a model selection criterion, most often the Bayesian information criterion. Rather than throw awa…
Finite mixture models are among the most popular statistical models used in different data science disciplines. Despite their broad applicability, inference under these models typically leads to computationally challenging non-convex problems. While the Expectation-Maximization (EM) algorithm is the most popular approa…
A new EM gradient algorithm for mixture models with skewed components.
problem Fitting mixture models with skewed components derived from the Manly transformation.
method Proposes an alternative EM gradient algorithm using Newton's method for better parameter updates.
result Shows improved convergence and parameter estimation compared to the Nelder-Mead optimization.
Finite mixture models are statistical models which appear in many problems in statistics and machine learning. In such models it is assumed that data are drawn from random probability measures, called mixture components, which are themselves drawn from a probability measure P over probability measures. When estimating …
CDNs improve uncertainty quantification in NNs with adaptive mixture components.
problem Uncertainty quantification in deep neural networks.
method Compound Density Networks (CDNs) combining Bayesian and ensemble approaches.
result CDNs provide better uncertainty estimates and robustness to adversarial examples.
There has been great interest recently in applying nonparametric kernel mixtures in a hierarchical manner to model multiple related data samples jointly. In such settings several data features are commonly present: (i) the related samples often share some, if not all, of the mixture components but with differing weight…
New method reduces mixture model evaluation cost for large models.
problem Computational infeasibility of evaluating all mixture components.
method Combining EM and Metropolis-Hastings for stochastic sampling.
result Significantly reduced computational cost for large models.
Motivated by generating personalized recommendations using ordinal (or preference) data, we study the question of learning a mixture of MultiNomial Logit (MNL) model, a parameterized class of distributions over permutations, from partial ordinal or preference data (e.g. pair-wise comparisons). Despite its long standing…
Two approaches improve parameter learning in various mixture models.
problem Parameter learning in mixture models.
method Complex-analytic and algebraic-combinatorial methods.
result Improved sample sufficiency for parameter estimation in specific mixture models.
Mixture components improve VAE performance by increasing latent flexibility.
problem Improving variational autoencoder (VAE) performance through more flexible latent representations.
method Modeling mixture components with separate encoder networks and analyzing their impact on ELBO.
result Increasing the number of mixture components improves VAE performance on various datasets.
Regression mixture models are widely studied in statistics, machine learning and data analysis. Fitting regression mixtures is challenging and is usually performed by maximum likelihood by using the expectation-maximization (EM) algorithm. However, it is well-known that the initialization is crucial for EM. If the init…
Algorithm estimates nonparametric mixtures from grouped data.
problem Estimating identifiable nonparametric mixture models from grouped observations.
method Oracle inequality for weighted kernel density estimators and general consistency result.
result Consistent estimation of mixture components from grouped observations.
This paper tackles causal interactions in mixtures of DAGs using interventions.
problem Learning causal interactions among variables governed by a mixture of causal systems.
method Establishes necessary and sufficient conditions for intervention size, designs an adaptive algorithm.
result Identifies true edges in a mixture of DAGs using optimal or near-optimal interventions.
We address two shortcomings in online travel time estimation methods for congested urban traffic. The first shortcoming is related to the determination of the number of mixture modes, which can change dynamically, within day and from day to day. The second shortcoming is the wide-spread use of Gaussian probability dens…
Study improves choice model accuracy and heterogeneity representation using mixture models.
problem Improving prediction accuracy and heterogeneity representation in choice models.
method Semi-nonparametric Latent Class Choice Model with mixture models and EM algorithm.
result Mixture models enhance prediction accuracy and heterogeneity representation without sacrificing interpretability.
In many applications, a finite mixture is a natural model, but it can be difficult to choose an appropriate number of components. To circumvent this choice, investigators are increasingly turning to Dirichlet process mixtures (DPMs), and Pitman-Yor process mixtures (PYMs), more generally. While these models may be well…
Mixtures of Mallows models are a popular generative model for ranking data coming from a heterogeneous population. They have a variety of applications including social choice, recommendation systems and natural language processing. Here we give the first polynomial time algorithm for provably learning the parameters of…
ELU algorithm improves on EM for over-specified Gaussian mixtures.
problem Slow convergence of EM in over-specified Gaussian mixtures.
method Developed ELU algorithm for two-component mixtures, combining exponential location update and gradient descent.
result ELU converges to final statistical radius after logarithmic iterations, resolving open question.
The Expectation-Maximization algorithm is perhaps the most broadly used algorithm for inference of latent variable problems. A theoretical understanding of its performance, however, largely remains lacking. Recent results established that EM enjoys global convergence for Gaussian Mixture Models. For Mixed Linear Regres…
One-bit clustering method for two-component sub-Gaussian mixture models
problem Clustering in sub-Gaussian mixture models
method One-bit clustering using dithered quantization
result Decaying misclassification rate with exponential signal-to-noise ratio
Naive Bayes spam filters are highly susceptible to data poisoning attacks. Here, known spam sources/blacklisted IPs exploit the fact that their received emails will be treated as (ground truth) labeled spam examples, and used for classifier training (or re-training). The attacking source thus generates emails that will…
Continuous-time interpolation of volatility surfaces preserving mixtures and arbitrage-free.
problem Interpolation of volatility surfaces
method Constructing a mixture-preserving, arbitrage-free interpolation
result Lifts Brigo-Mercurio to time-varying weights with additive cost
The resolution and calibration of pure spectra of minority components in measurements of chemical mixtures without prior knowledge of the mixture is a challenging problem. In this work, a combination of band target entropy minimization (BTEM) and target partial least squares (T-PLS) was used to obtain estimates for sin…
Discover causal structure from mixtures of DAGs using latent variable algorithms.
problem Discover causal structure from distributions arising from mixtures of DAGs.
method Causal structure discovery algorithms such as FCI for latent variables.
result Recover a 'union' of the component DAGs and identify varying conditional distributions.
Proposes a method to decompose multivariate signals into Gaussian components.
problem Decomposing multivariate signals into Gaussian components.
method Greedy variational method for non-negative multivariate signals as a weighted sum of Gaussians.
result Upper bound for the distance from any mode of a Gaussian mixture model to the set of corresponding means.
LSTM-MDNs improve risk forecasting during turbulent periods.
problem Forecasting Value-at-Risk (VaR) during volatile market conditions.
method Implemented Long Short-Term Memory mixture density networks (LSTM-MDNs) for VaR forecasting and compared them with established models.
result LSTM-MDNs outperformed benchmark models in turbulent periods but not in calm periods.
Proposes a new model for mixed membership in Gaussian mixture.
problem Limited to single component membership in Gaussian mixture models.
method Mixed membership sub-Gaussian model, spectral algorithm.
result Estimation error can be made arbitrarily small with high probability.
A family of parsimonious shifted asymmetric Laplace mixture models is introduced. We extend the mixture of factor analyzers model to the shifted asymmetric Laplace distribution. Imposing constraints on the constitute parts of the resulting decomposed component scale matrices leads to a family of parsimonious models. An…
Essential principal components simplify spectral analysis with minimal training data.
problem Accurate spectral quantification from complex mixtures.
method Identifying essential principal components and using molar extinction coefficients.
result Near one-to-one projection from principal components to mixture constituents.