Robust learning mixtures of linear regressions improve robustness.
problem Improving robustness in learning mixtures of linear regressions.
method Connecting mixtures of linear regressions and mixtures of Gaussians with thresholding for a quasi-polynomial time algorithm.
result The algorithm has significantly better robustness than previous results.
In recent years, correntropy and its applications in machine learning have been drawing continuous attention owing to its merits in dealing with non-Gaussian noise and outliers. However, theoretical understanding of correntropy, especially in the statistical learning context, is still limited. In this study, within the…
Extends Gaussian Process regression for handling multiple prior distributions.
problem Handling multiple prior distributions in Bayesian Machine Learning models.
method Mixtures of Gaussian Processes with analytical and Sparse Variational approaches.
result Effective in accounting for prior misspecification in functional regression problems.
Study uniform consistency in nonparametric mixture models and mixed regression.
problem Uniform consistency in nonparametric mixture models and mixed regression models.
method Construct uniformly consistent estimators under general conditions, develop novel technical tools.
result Prove uniform consistency results for nonparametric mixtures and mixed regression models.
GGMPs improve non-Gaussian conditional density estimation.
problem Multimodality, heteroscedasticity, and strong non-Gaussianity in conditional density estimation.
method GGMP combines local Gaussian mixture fitting, cross-input component alignment, and per-component heteroscedastic GP training.
result GGMPs improve distributional approximation on synthetic and real-world datasets.
EM algorithm converges globally for two-component mixed linear regression.
problem Global convergence of EM algorithm for mixed linear regression.
method Developed new theoretical analysis for EM algorithm convergence in mixed linear regression.
result EM algorithm converges globally for two-component mixed linear regression.
This paper presents a new model called infinite mixtures of multivariate Gaussian processes, which can be used to learn vector-valued functions and applied to multitask learning. As an extension of the single multivariate Gaussian process, the mixture model has the advantages of modeling multimodal data and alleviating…
Algorithm learns mixtures of Gaussians efficiently using diffusion models.
problem Learning mixtures of Gaussians with identity covariance.
method Analytic approach using diffusion models to learn score functions.
result Quasi-polynomial time and sample complexity for learning mixtures.
We propose a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression. Our mixture-of-experts model is conceptually simple and hierarchically recombines computations for an overall approximation of a full Gaussian process. Closed-form and distributed computations allow for effici…
Proposes EMG mixture model for spectroscopy data.
problem Modeling residuals in spectroscopy data with positive support.
method Exponentially-modified Gaussian mixture (EMG) model with expectation-maximization algorithm.
result EMG mixture outperforms existing models in spectroscopy applications.
Model-based clustering approaches concern the paradigm of exploratory data analysis relying on the finite mixture model to automatically find a latent structure governing observed data. They are one of the most popular and successful approaches in cluster analysis. The mixture density estimation is generally performed …
Training Gaussian process-based models typically involves an O(N3) computational bottleneck due to inverting the covariance matrix. Popular methods for overcoming this matrix inversion problem cannot adequately model all types of latent functions, and are often not parallelizable. However, judicious choice of model…
This paper introduces a novel model-based clustering approach for clustering time series which present changes in regime. It consists of a mixture of polynomial regressions governed by hidden Markov chains. The underlying hidden process for each cluster activates successively several polynomial regimes during time. The…
Mixture models with Gamma and or inverse-Gamma distributed mixture components are useful for medical image tissue segmentation or as post-hoc models for regression coefficients obtained from linear regression within a Generalised Linear Modeling framework (GLM), used in this case to separate stochastic (Gaussian) noise…
This paper learns motion primitives from driving data to improve vehicle path-tracking.
problem Improving vehicle path-tracking accuracy through learned motion primitives.
method Two-level structure with path segmentation and clustering; Gaussian Mixture Model (GMM) and Gaussian Mixture Regression (GMR).
result The model predicts future lateral control commands with high accuracy.
The paper optimizes hyperplanes for binary classification in high-dimensional data with latent Gaussian mixtures.
problem Binary classification in high-dimensional data with latent Gaussian mixtures.
method Generalized least squares estimator for estimating the direction of the optimal separating hyperplane. Simple correction for intercept estimation.
result The procedure is minimax optimal in many scenarios and can retain the interpolation property.
We propose Dirichlet Process mixtures of Generalized Linear Models (DP-GLM), a new method of nonparametric regression that accommodates continuous and categorical inputs, and responses that can be modeled by a generalized linear model. We prove conditions for the asymptotic unbiasedness of the DP-GLM regression mean fu…
In this paper we propose a novel framework for the construction of sparsity-inducing priors. In particular, we define such priors as a mixture of exponential power distributions with a generalized inverse Gaussian density (EP-GIG). EP-GIG is a variant of generalized hyperbolic distributions, and the special cases inclu…
Study of two-layer NNs under Gaussian mixtures data, proving polynomial models equivalent to neural networks.
problem Training and generalization performance of two-layer NNs under structured Gaussian mixture data.
method Asymptotic analysis of two-layer NNs after one gradient descent step under Gaussian mixture data assumption.
result High-order polynomial models equivalent to nonlinear neural networks under certain conditions.
Paper introduces deep structured mixtures of Gaussian processes for scalable GP approximations.
problem Scalability issues with Gaussian Processes (GPs).
method Deep structured mixtures of GP experts for scalable approximate inference.
result Deep structured mixtures provide better predictive uncertainties and competitive performance.
A novel Bayesian method for dynamic sparsity in Gaussian dynamic linear regression.
problem Variable selection and shrinkage in time-varying regression models.
method Time-varying sparsity via Markov switching priors for coefficients' variances, extending spike-and-slab priors.
result Induces smoothness or shrinkage towards zero at each time point, leading to improved model performance.
A new model predicts multivariate regression using similarities to data points.
problem Complex, high-dimensional input-output relationships.
method Bayesian mixture-of-experts with conditional Gaussian mixtures and variational Bayes.
result Outperforms competitors in high-dimensional settings.
Method for initializing Gaussian mixtures for variational inference with multi-modal distributions.
problem Challenges in variational inference with Gaussian mixtures due to multimodality and nonconvex loss functions.
method Optimization to find local maxima, local Gaussian approximations, and constrained least squares regression.
result Robust initialization improves variational inference performance and scalability.
Proposes scale mixture of NNGPs for more flexible stochastic processes.
problem Limited focus on broadening the class of stochastic processes from NNGPs.
method Scale mixture of NNGPs with scale priors on last-layer parameters.
result Turns neural networks into a richer class of stochastic processes.
We study high-dimensional Gaussian mixture classification using statistical physics methods.
problem Classifying high-dimensional Gaussian mixture with general covariance matrices.
method Replica method from statistical physics for asymptotic analysis of convex classifiers.
result Construction and validation of a de-biased estimator for variable selection.
Paper proposes a novel method to accurately determine the number of experts in Gaussian-gated Gaussian MoE models.
problem Challenges in model selection for MoE models, especially with covariates.
method Introduces a novel extension using dendrograms of mixing measures to estimate the true number of mixture components.
result Achieves optimal convergence rates for parameter estimation and accurately approximates the regression function.
Splat Regression Models use mixtures of bump functions to approximate complex data.
problem Approximating complex data with high interpretability and accuracy.
method Model outputs are mixtures of heterogeneous and anisotropic bump functions (splats) weighted by output vectors. Fitting splat models reduces to optimization over mixing measures using Wasserstein-Fisher-Rao gradient flows.
result Unified theoretical framework for Gaussian Splatting and flexible approach for diverse problems.
This paper studies generalization in machine learning with mixture data.
problem Generalization performance and statistical rates in heterogeneous data.
method Characterization of heterogeneity via pairwise total variation distance, analysis of Rademacher and Gaussian complexities.
result The requirement on heterogeneity increases as function classes get more complex.
Efficient algorithms for sparse parameter recovery in mixture models.
problem Support recovery of high-dimensional sparse latent vectors in mixture models.
method Efficient algorithms with logarithmic sample complexity dependence on dimensionality.
result First guarantees on support recovery for various mixture models.
Study shows how over-parameterized classifiers can still perform well on noisy data.
problem Understanding how maximum margin classifiers perform in over-parameterized settings with noisy data.
method Analyzes maximum margin classifiers on sub-Gaussian mixtures, providing risk bounds.
result Characterizes conditions for 'benign overfitting' in linear classification problems.
Regularization helps improve classification of noisy high-dimensional data.
problem Classifying high-dimensional noisy Gaussian mixture with limited oracle knowledge.
method Analysis of regularized convex classifiers including ridge, hinge, and logistic regression.
result Regularization can reach Bayes-optimal performance under certain conditions.
New approach for distributed learning of Gaussian mixtures.
problem Large datasets distributed across different centers.
method Split-and-conquer approach with MM algorithm.
result New estimator is consistent and retains root-n consistency.
Elliptical processes extend Gaussian models with heavier tails.
problem Regression and classification with non-Gaussian likelihoods or heavy tails.
method Spline normalizing flow for variational inference of elliptical distributions.
result Elliptical processes outperform Gaussian processes in non-Gaussian settings.
In this work we introduce a mixture of GPs to address the data association problem, i.e. to label a group of observations according to the sources that generated them. Unlike several previously proposed GP mixtures, the novel mixture has the distinct characteristic of using no gating function to determine the associati…
Generalized autoregressive conditional heteroscedasticity (GARCH) models have long been considered as one of the most successful families of approaches for volatility modeling in financial return series. In this paper, we propose an alternative approach based on methodologies widely used in the field of statistical mac…
The LASSO is a recent technique for variable selection in the regression model \bean y & = & Xβ+ z, \eean where X∈Rn×p and z is a centered gaussian i.i.d. noise vector N(0,σ2I). The LASSO has been proved to achieve remarkable properties such as exact support recovery of sparse vectors when …
VBphenoR uses variational Bayes for EHR-based patient phenotyping.
problem Phenotyping patients from EHR data for targeted treatments.
method Variational Bayes Gaussian Mixture Model (GMM) and logistic regression.
result Closed-form inference for efficient patient phenotype determination.
Improved VB algorithm for NIG mixtures outperforms Gaussian mixtures for non-Gaussian data.
problem Clustering non-Gaussian data, especially heavy-tailed and asymmetric.
method Proposed an improved VB algorithm for NIG mixture models and extended Dirichlet process mixture models.
result Outperforms Gaussian mixtures and existing NIG mixture models, especially for highly non-normative data.
Elliptical processes generalize Gaussian and Student-t models with fat tails and computational efficiency.
problem Need for models with fat tails and computational tractability.
method Represent elliptical distributions as continuous mixtures of Gaussian distributions, derive closed-form expressions for marginal and conditional distributions.
result Elliptical processes offer advantages in robust regression compared to Gaussian processes.
Method estimates mixture components without discretizing parameters.
problem Learning from mixtures of continuous features with noise.
method Off-the-grid optimization method for continuous parameter space.
result Prediction error bound similar to Lasso predictor.
A fast method combines deep mixtures of sparse GPs for flexible modeling.
problem Flexible modeling with changing output densities.
method Designing gating network with DNN for selecting sparse GPs, using CCR algorithm.
result The method outperforms competing methods in accuracy and uncertainty quantification.
New DP EM algorithm with statistical guarantees for mixture models.
problem Preserving privacy in EM algorithms for mixture models.
method Proposed a DP EM algorithm with statistical guarantees.
result Near optimal estimation error for GMM in DP model.
MM (majorization--minimization) algorithms are an increasingly popular tool for solving optimization problems in machine learning and statistical estimation. This article introduces the MM algorithm framework in general and via three popular example applications: Gaussian mixture regressions, multinomial logistic regre…
Algorithm recovers multiple low-rank matrices from unlabeled data.
problem Learning mixtures of low-rank models from unlabelled data.
method Three-stage meta-algorithm that copes with non-convexity and noise.
result Near-optimal sample and computational complexities under Gaussian designs.
Algorithm identifies and corrects noisy labels using Gaussian process regression.
problem Detecting and correcting real-valued noisy labels from mixed data.
method Gaussian process regression with heteroscedastic noise model and leave-one-out cross-validation.
result The method can pinpoint corrupted sample points and improve regression models.
New Gaussian min-max theorem extends classical results to non-i.i.d. Gaussian matrices.
problem Extending classical Gaussian min-max theorems to non-i.i.d. Gaussian matrices.
method Identifying a new pair of Gaussian processes that satisfy comparison inequalities.
result New Gaussian min-max and convex Gaussian min-max theorems with applications in multi-source Gaussian regression and binary classification.
The paper uses Gaussian mixture models for Bayesian networks and proposes an optimization algorithm.
problem Modeling nodes in Bayesian networks with complex distributions.
method Gaussian mixture models combined with double iteration algorithm.
result The double iteration algorithm optimizes Gaussian mixture models effectively.
The study bounds the stability of Gaussian mixtures under small perturbations.
problem Stability of Gaussian mixtures under small changes in distribution.
method Deriving an explicit bound on parameter stability of spherical Gaussian Mixture Models (sGMM) in a pre-defined model class.
result Upper bound on parameter distance of close sGMMs to the original sGMM, dependent only on the original model.