arXiv research

Robust learning mixtures of linear regressions improve robustness.

Two EM algorithms estimate prior distributions in mixture of linear regressions.

Study on recovering supports of multiple sparse vectors from mixed linear measurements.

The Expectation-Maximization algorithm is perhaps the most broadly used algorithm for inference of latent variable problems. A theoretical understanding of its performance, however, largely remains lacking. Recent results established that EM enjoys global convergence for Gaussian Mixture Models. For Mixed Linear Regres…

Gradient EM converges exponentially to optimal solution in agnostic mixtures.

We propose Dirichlet Process mixtures of Generalized Linear Models (DP-GLM), a new method of nonparametric regression that accommodates continuous and categorical inputs, and responses that can be modeled by a generalized linear model. We prove conditions for the asymptotic unbiasedness of the DP-GLM regression mean fu…

Mix-IRLS solves imbalanced mixed linear regression problems efficiently.

Wasserstein framework solves mixed linear regression problems.

Discriminative latent-variable models are typically learned using EM or gradient-based optimization, which suffer from local optima. In this paper, we develop a new computationally efficient and provably consistent estimator for a mixture of linear regressions, a simple instance of a discriminative latent-variable mode…

Transformers can learn optimal regression mixtures efficiently.

This paper studies generalization in machine learning with mixture data.

Transformers can learn mixture of linear models efficiently.

Estimates MLDS using tensor decomposition, improving upon existing methods.

The paper improves EM for clustering in a mixture of linear regression models.

Flexible empirical Bayes for large-scale multiple linear regression.

New model handles complex non-linear relationships with hidden graph structures.

We introduce a convex approach for mixed linear regression over $d$ features. This approach is a second-order cone program, based on L1 minimization, which assigns an estimate regression coefficient in $\mathbb{R}^{d}$ for each data point. These estimates can then be clustered using, for example, $k$-means. For problem…

Efficient algorithms for sparse parameter recovery in mixture models.

New AMP algorithm estimates signals and latent variables in mixed regression models.

Paper tackles MLR prediction error without assuming realizable models.

In this paper, we study the modeling and the classification of functional data presenting regime changes over time. We propose a new model-based functional mixture discriminant analysis approach based on a specific hidden process regression model that governs the regime changes over time. Our approach is particularly a…

The paper develops methods to create reliable prediction sets for complex mixture models in high-dimensional data.

We study the convergence of the Expectation-Maximization (EM) algorithm for mixtures of linear regressions with an arbitrary number $k$ of components. We show that as long as signal-to-noise ratio (SNR) is $\tildeΩ(k)$, well-initialized EM converges to the true regression parameters. Previous results for $k \geq 3$ hav…

A novel Bayesian method for dynamic sparsity in Gaussian dynamic linear regression.

We give convergence guarantees for estimating the coefficients of a symmetric mixture of two linear regressions by expectation maximization (EM). In particular, we show that the empirical EM iterates converge to the target parameter vector at the parametric rate, provided the algorithm is initialized in an unbounded co…

Mixtures-of-Experts (MoE) are conditional mixture models that have shown their performance in modeling heterogeneity in data in many statistical learning approaches for prediction, including regression and classification, as well as for clustering. Their estimation in high-dimensional problems is still however challeng…

We consider the problem of learning a mixture of linear regressions (MLRs). An MLR is specified by $k$ nonnegative mixing weights $p_1, \ldots, p_k$ summing to $1$, and $k$ unknown regressors $w_1,...,w_k\in\mathbb{R}^d$. A sample from the MLR is drawn by sampling $i$ with probability $p_i$, then outputting $(x, y)$ wh…

We consider a discriminative learning (regression) problem, whereby the regression function is a convex combination of k linear classifiers. Existing approaches are based on the EM algorithm, or similar techniques, without provable guarantees. We develop a simple method based on spectral techniques and a `mirroring' tr…

Study shows how over-parameterized classifiers can still perform well on noisy data.

We propose a novel exponentially-modified Gaussian (EMG) mixture residual model. The EMG mixture is well suited to model residuals that are contaminated by a distribution with positive support. This is in contrast to commonly used robust residual models, like the Huber loss or $\ell_1$, which assume a symmetric contami…

Study uniform consistency in nonparametric mixture models and mixed regression.

Machine learning improves joint default assessment by capturing non-linear dependencies.

fMRI semantic category understanding using linguistic encoding models attempts to learn a forward mapping that relates stimuli to the corresponding brain activation. State-of-the-art encoding models use a single global model (linear or non-linear) to predict brain activation given the stimulus. However, the critical as…

Method estimates mixture components without discretizing parameters.

Efficient algorithms recover two sparse models from a mix of linear queries.

NMDR estimates complex mixtures of distributions efficiently.

Mixture of Experts (MoE) is a popular framework in the fields of statistics and machine learning for modeling heterogeneity in data for regression, classification and clustering. MoE for continuous data are usually based on the normal distribution. However, it is known that for data with asymmetric behavior, heavy tail…

New DP EM algorithm with statistical guarantees for mixture models.

Finite mixtures of regression models offer a flexible framework for investigating heterogeneity in data with functional dependencies. These models can be conveniently used for unsupervised learning on data with clear regression relationships. We extend such models by imposing an eigen-decomposition on the multivariate …

Mixture models with Gamma and or inverse-Gamma distributed mixture components are useful for medical image tissue segmentation or as post-hoc models for regression coefficients obtained from linear regression within a Generalised Linear Modeling framework (GLM), used in this case to separate stochastic (Gaussian) noise…

Statistical approaches for Functional Data Analysis concern the paradigm for which the individuals are functions or curves rather than finite dimensional vectors. In this paper, we particularly focus on the modeling and the classification of functional data which are temporal curves presenting regime changes over time.…

A method for identifying NPWARX models with arbitrary domains using probabilistic mixture models.

Robust Bayesian models are appealing alternatives to standard models, providing protection from data that contains outliers or other departures from the model assumptions. Historically, robust models were mostly developed on a case-by-case basis; examples include robust linear regression, robust mixture models, and bur…

EM algorithm converges linearly and achieves sharp rate in estimating mixtures of pairwise differences.

AM converges super-linearly for solving mixed linear regression problems.

A new clustering method for functional data using skewed distributions.

Algorithm recovers multiple low-rank matrices from unlabeled data.

In the problem of learning mixtures of linear regressions, the goal is to learn a collection of signal vectors from a sequence of (possibly noisy) linear measurements, where each measurement is evaluated on an unknown signal drawn uniformly from this collection. This setting is quite expressive and has been studied bot…