Paper introduces deep structured mixtures of Gaussian processes for scalable GP approximations.
problem Scalability issues with Gaussian Processes (GPs).
method Deep structured mixtures of GP experts for scalable approximate inference.
result Deep structured mixtures provide better predictive uncertainties and competitive performance.
Method for initializing Gaussian mixtures for variational inference with multi-modal distributions.
problem Challenges in variational inference with Gaussian mixtures due to multimodality and nonconvex loss functions.
method Optimization to find local maxima, local Gaussian approximations, and constrained least squares regression.
result Robust initialization improves variational inference performance and scalability.
Study finds the minimum number of finite Gaussian mixtures for best approximation.
problem Finding the minimum number of finite Gaussian mixtures for best approximation.
method Local moment matching for upper bound and spectral analysis for lower bound.
result Corrects a previous lower bound in the case of Gaussian mixing distributions.
We consider unsupervised estimation of mixtures of discrete graphical models, where the class variable corresponding to the mixture components is hidden and each mixture component over the observed variables can have a potentially different Markov graph structure and parameters. We propose a novel approach for estimati…
Study reveals structure of local minima in GMMs, identifying key cluster centers.
problem Identifying optimal cluster centers in non-convex GMM landscapes.
method Analyzing the negative log-likelihood function of GMMs in the population limit.
result Local minima share a common structure that partially identifies true cluster centers.
The paper explores methods for inference in multiplicative latent force models.
problem Inference in hybrid models combining mechanistic and flexible components.
method Two methods of approximate inference: gradient matching and mixtures of local approximations.
result Comparison of methods on simulated and motion capture data.
Paper proposes a privacy-preserving RL algorithm for linear MDPs with theoretical guarantees.
problem Protecting users' private data in personalized services using RL.
method Local differential privacy (LDP) for RL with linear function approximation.
result Achieves a regret bound of $O(d^{5/4}H^{7/4}T^{3/4}\left(\log(1/δ)
ight)^{1/4}\sqrt{1/\varepsilon})$ for linear mixture MDPs.
A new method improves posterior approximation for complex distributions.
problem Difficulty in capturing multimodal and heavy-tailed posteriors with standard normalizing flows.
method StiCTAF: stick-breaking mixture base with component-wise tail adaptation.
result Improved tail recovery and better mode coverage compared to benchmarks.
Efficiently trains GMMs for streaming data with SGD, addressing local optima and numerical instabilities.
problem Local optima and numerical instabilities in training GMMs for high-dimensional streaming data.
method Stochastic Gradient Descent (SGD) with adaptive annealing and exponential-free approximation.
result SGD approach trains GMMs without k-means initialization and outperforms sEM for high-dimensional data.
New sampling and identity-testing methods for mixtures of distributions that don't satisfy approximate tensorization of entropy.
problem Sampling and identity-testing for mixtures of distributions that don't satisfy approximate tensorization of entropy.
method Fast mixing of Glauber dynamics and efficient identity-testers in the coordinate-conditional sampling access model.
result Efficient identity-testers for mixtures of ATE distributions in the coordinate-conditional sampling access model.
Improves posterior approximation speed for Dirichlet process mixture models.
problem Inefficiency of stochastic variational inference in large datasets.
method Uses stochastic gradient ascent with adaptive stepsize optimization.
result Adaptive stepsize improves speed and performance of posterior approximation.
Bayesian mixture models inference in federated learning
problem Inference of Bayesian mixture models in federated learning
method Consensus Monte Carlo approach
result Recovery of small clusters with greater accuracy than standard MCMC
Theoretical analysis of entropy approximation for Gaussian mixtures.
problem Lack of theoretical guarantees for entropy approximation of Gaussian mixtures.
method Theoretical analysis of the error between true and approximate entropy.
result The error converges to zero as the ratios of means to variances tend to infinity, providing a guarantee for high-dimensional problems.
GGMPs improve non-Gaussian conditional density estimation.
problem Multimodality, heteroscedasticity, and strong non-Gaussianity in conditional density estimation.
method GGMP combines local Gaussian mixture fitting, cross-input component alignment, and per-component heteroscedastic GP training.
result GGMPs improve distributional approximation on synthetic and real-world datasets.
Most models for barrier pricing are designed to let a market maker tune the model-implied covariance between moves in the asset spot price and moves in the implied volatility skew. This is often implemented with a local volatility/stochastic volatility mixture model, where the mixture parameter tunes that covariance. T…
A fast method combines deep mixtures of sparse GPs for flexible modeling.
problem Flexible modeling with changing output densities.
method Designing gating network with DNN for selecting sparse GPs, using CCR algorithm.
result The method outperforms competing methods in accuracy and uncertainty quantification.
Variational Bayes (VB), also known as independent mean-field approximation, has become a popular method for Bayesian network inference in recent years. Its application is vast, e.g. in neural network, compressed sensing, clustering, etc. to name just a few. In this paper, the independence constraint in VB will be relax…
Mixtures of neural operators reduce active complexity in operator learning.
problem Reduction of active complexity in operator learning models.
method Constructive comparison between routed mixtures of neural operators (MoNOs) and a fixed single-neural-operator construction.
result Every scalar uniformly continuous nonlinear operator can be approximated by a MoNO whose active expert has smaller depth, width, and rank scaling.
This paper compares EM and GD in two-component mixture models, finding EM escapes bad local optima more reliably.
problem Understanding the convergence of EM and GD in mixture models, especially in regions where one component is missing.
method Analyzing regions called one-cluster regions in two-component mixture models of Gaussians and Bernoullis, comparing the propensity of EM and GD to converge to these regions.
result EM escapes one-cluster regions exponentially fast, while GD escapes them linearly fast, indicating EM is less likely to converge to bad local optima.
Splat Regression Models use mixtures of bump functions to approximate complex data.
problem Approximating complex data with high interpretability and accuracy.
method Model outputs are mixtures of heterogeneous and anisotropic bump functions (splats) weighted by output vectors. Fitting splat models reduces to optimization over mixing measures using Wasserstein-Fisher-Rao gradient flows.
result Unified theoretical framework for Gaussian Splatting and flexible approach for diverse problems.
New method uses subtractive mixture models for approximate inference.
problem How to effectively use subtractive mixture models for approximate inference.
method Design expectation estimators for IS and learning schemes for VI with SMMs.
result Empirical evaluation shows SMMs can approximate distributions effectively.
This work links SOMs and GMMs, providing a mathematical basis for their use.
problem Understanding the relationship between SOMs and GMMs.
method Mathematical treatment showing SOMs as gradient descent on a GMM log-likelihood.
result SOMs can be interpreted as probabilistic models, justifying their use in various applications.
Parameter estimation for model-based clustering using a finite mixture of normal inverse Gaussian (NIG) distributions is achieved through variational Bayes approximations. Univariate NIG mixtures and multivariate NIG mixtures are considered. The use of variational Bayes approximations here is a substantial departure fr…
Improved phylogenetic inference using VBPI-Mixtures for tree topology and branch length.
problem Multimodality of tree-topology posterior distributions in phylogenetic inference.
method VBPI-Mixtures algorithm that uses mixture learning within the BBVI framework.
result VBPI-Mixtures captures tree-topology distributions better than VBPI.
Conditional diffusion models can approximate target distributions well with Gaussian-mixture reverse kernels.
problem Approximating target distributions in conditional diffusion models.
method Using finite Gaussian mixtures with ReLU-network logits as reverse kernels, reducing the problem to static conditional density approximation.
result The resulting neural reverse-kernel class is dense in conditional KL divergence under exact terminal matching.
Local asymptotic minimax risk bounds in a locally asymptotically mixture of normal family of distributions have been investigated under asymmetric loss functions and the asymptotic distribution of the optimal estimator that attains the bound has been obtained.
The study tightens risk bounds for mixtures of experts using local differential privacy.
problem Improving risk bounds for mixtures of experts.
method Imposing local differential privacy (LDP) on the gating mechanism of mixtures of experts.
result Theoretical bounds exhibit logarithmic dependence on the number of experts and tighter than existing bounds.
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using …
Gradient method converges locally linearly for overparameterized Gaussian mixtures.
problem Learning Gaussian mixtures under overparameterization.
method Gradient-based method alternating short descent steps and long Polyak steps.
result Gradient method converges locally linearly to minimizers.
Develops a framework for distilling flow models from few steps.
problem Improving few-step sampling in diffusion models for better performance.
method Local approximation errors and dynamical amplification controlled through analytical tractability.
result Deep residual compositions efficiently approximate long-horizon transport with controlled global error.
We extend natural-gradient methods to mixtures of exponential-family distributions, improving inference speed.
problem Complex, multimodal posterior distributions are difficult to approximate with simple exponential-family distributions.
method We use minimal conditional-EF representations and derive simple natural-gradient updates.
result Our natural-gradient method converges faster than black-box methods with reparameterization gradients.
Discriminative classifier for compositional data using hierarchical mixture of Generalized Dirichlet models.
problem Classifying compositional data, especially in spam detection and color space identification.
method Hierarchical mixture of discriminative Generalized Dirichlet classifiers, using variational approximation for parameter learning.
result First time a variational upper-bound for Generalized Dirichlet mixture is proposed in literature.
SFSVI uses Gaussian mixtures to approximate neural network outputs for continual learning.
problem Learning new tasks without forgetting old ones in neural networks.
method Sequential function-space variational inference with Gaussian mixture approximation.
result Gaussian mixture SFSVI outperforms other methods in continual learning.
A new probabilistic polygonal curve representation using Gaussian Mixture Models.
problem Capturing curves with uncertainty in both tangent and normal directions.
method Probabilistic polygonal approximation with Gaussian Mixture Model (GMM).
result The GMM accurately captures the local geometry and uncertainty of curves.
Learning representation on graph plays a crucial role in numerous tasks of pattern recognition. Different from grid-shaped images/videos, on which local convolution kernels can be lattices, however, graphs are fully coordinate-free on vertices and edges. In this work, we propose a Gaussian-induced convolution (GIC) fra…
DFMR improves robustness of learning finite mixture models in distributed settings.
problem Learning finite mixture models in distributed settings with Byzantine failures.
method DFMR leverages pairwise L2 distances to filter and retain local estimates, ensuring robust aggregation.
result DFMR achieves optimal convergence rate and asymptotic equivalence to global maximum likelihood estimate.
Bayesian approach approximates probability functions of Gaussian mixtures.
problem Approximating probability functions of non-spherical Gaussian mixtures.
method Bayesian decomposition, spherical radial decomposition, random sampling.
result Established differentiability and integral representation of gradient for probability functions.
This study analyzes theoretical guarantees for VI with fixed-variance Gaussian mixtures.
problem Theoretical analysis of variational inference with non-Gaussian distributions.
method Formulates variational inference as minimizing a mollified relative entropy, solving it through gradient descent on particle positions.
result Establishes descent lemma and approximation error bounds for optimization of variational inference.
Paper proposes new costs for learning multiple centers in MDNs.
problem Learning multiple centers for density approximation in MDNs.
method Combines MDNs with contrastive costs using four types of kernelized matrix costs.
result New costs improve data density approximation in MDNs.
Paper analyzes approximation and learning of MoEs with (P)ReLU activation.
problem Scaling up deep learning models with MoEs and (P)ReLU activation.
method Approximation and learning-theoretic analysis of MoMLPs with (P)ReLU.
result MoMLPs can uniformly approximate Lipschitz functions with ε accuracy using O(ε−1) parameters. New method improves Gaussian Mixture Model fitting speed.
problem Slow convergence of EM algorithm for complex data.
method Riemannian Newton Trust-Region method for Gaussian Mixture Models.
result Outperforms existing methods in runtime and iterations.
A novel optimization-based Gaussian mixture reduction method using composite transportation divergence.
problem Exponential increase in Gaussian mixture order leads to intractable inference.
method Optimization-based Gaussian mixture reduction (GMR) using composite transportation divergence (CTD).
result Unified framework for selecting optimal cost function in various applications.
UBVI improves variational inference by preventing degeneracy and improving scalability.
problem Degeneracy and scalability issues in variational inference.
method Exploits Hellinger metric geometry to prevent degeneracy, simplifies weight optimization, and uses scalable exponential family mixture components.
result Output of UBVI converges to best possible approximation in any mixture family, even when misspecified.
Analytical method approximates ELBO gradient in clutter problem.
problem Clutter problem in Bayesian networks with Gaussian likelihood.
method Reparameterization trick, local approximation of likelihood factors.
result Good accuracy and linear computational complexity compared to classical methods.
The Hierarchical Mixture of Experts (HME) is a well-known tree-based model for regression and classification, based on soft probabilistic splits. In its original formulation it was trained by maximum likelihood, and is therefore prone to over-fitting. Furthermore the maximum likelihood framework offers no natural metri…
LI-ITR combines flexible ML with interpretable approximations for personalized treatment rules.
problem Combining flexibility and interpretability in personalized treatment rules.
method Uses variational autoencoders and a mixture of interpretable experts.
result Accurately recovers true local coefficients and optimal treatment strategies.
A new method uses mixture approximations to improve diffusion models for Bayesian inverse problems.
problem Approximating posterior distributions in Bayesian inverse problems with intractable likelihoods.
method Proposes a mixture-based approximation of intermediate posterior distributions and uses Gibbs sampling for practical sampling.
result Validated the approach on image inverse problems and audio source separation, demonstrating improved performance.
Neighbor Mixture Model captures node correlations in graphs.
problem Modeling correlations between node labels in graphs.
method Neighbor Mixture Model (NMM) designed for efficient computation and scalability.
result NMM outperforms state-of-the-art models in various graph tasks.