arXiv research

We derive both {\em local} and {\em global} generalized {\em Bianchi identities} for classical Lagrangian field theories on gauge-natural bundles. We show that globally defined generalized Bianchi identities can be found without the {\em a priori} introduction of a connection. The proof is based on a {\em global} decom…

Clarifies EM algorithm and variational Bayesian inference concepts.

One iteration of standard $k$-means (i.e., Lloyd's algorithm) or standard EM for Gaussian mixture models (GMMs) scales linearly with the number of clusters $C$, data points $N$, and data dimensionality $D$. In this study, we explore whether one iteration of $k$-means or EM for GMMs can scale sublinearly with $C$ at run…

In the classical Lagrangian approach to conservation laws of gauge-natural field theories a suitable (vector) density is known to generate the so--called {\em conserved Noether currents}. It turns out that along any section of the relevant gauge--natural bundle this density is the divergence of a skew--symmetric (tenso…

We derive a novel variational expectation maximization approach based on truncated posterior distributions. Truncated distributions are proportional to exact posteriors within subsets of a discrete state space and equal zero otherwise. The treatment of the distributions' subsets as variational parameters distinguishes …

This paper models time-series data with a mixture of Markov chains, automatically determining the number of components.

MLE and CVE are equivalent under exponential families, leading to faster and more stable EM algorithms.

When a gauge-natural invariant variational principle is assigned, to determine {\em canonical} covariant conservation laws, the vertical part of gauge-natural lifts of infinitesimal principal automorphisms -- defining infinitesimal variations of sections of gauge-natural bundles -- must satisfy generalized Jacobi equat…

New EM algorithm improves deep generative network training.

GFlowNet-EM learns complex latent variable models with discrete structures.

Survey of factor analysis, PCA, variational inference, and VAE.

In classical Hawkes process, the baseline intensity and triggering kernel are assumed to be a constant and parametric function respectively, which limits the model flexibility. To generalize it, we present a fully Bayesian nonparametric model, namely Gaussian process modulated Hawkes process and propose an EM-variation…

Inference and learning for probabilistic generative networks is often very challenging and typically prevents scalability to as large networks as used for deep discriminative approaches. To obtain efficiently trainable, large-scale and well performing generative networks for semi-supervised learning, we here combine tw…

We present a family of expectation-maximization (EM) algorithms for binary and negative-binomial logistic regression, drawing a sharp connection with the variational-Bayes algorithm of Jaakkola and Jordan (2000). Indeed, our results allow a version of this variational-Bayes approach to be re-interpreted as a true EM al…

Marginal MAP problems are notoriously difficult tasks for graphical models. We derive a general variational framework for solving marginal MAP problems, in which we apply analogues of the Bethe, tree-reweighted, and mean field approximations. We then derive a "mixed" message passing algorithm and a convergent alternati…

We study inference and learning based on a sparse coding model with `spike-and-slab' prior. As in standard sparse coding, the model used assumes independent latent sources that linearly combine to generate data points. However, instead of using a standard sparse prior such as a Laplace distribution, we study the applic…

Training deep generative models with maximum likelihood remains a challenge. The typical workaround is to use variational inference (VI) and maximize a lower bound to the log marginal likelihood of the data. Variational auto-encoders (VAEs) adopt this approach. They further amortize the cost of inference by using a rec…

The Expectation-Maximization (EM) algorithm for mixture models often results in slow or invalid convergence. The popular convergence proof affirms that the likelihood increases with Q; Q is increasing in the M -step and non-decreasing in the E-step. The author found that (1) Q may and should decrease in some E-steps; (…

The speed of convergence of the Expectation Maximization (EM) algorithm for Gaussian mixture model fitting is known to be dependent on the amount of overlap among the mixture components. In this paper, we study the impact of mixing coefficients on the convergence of EM. We show that when the mixture components exhibit …

Cryo-electron microscopy (cryo-EM) is capable of producing reconstructed 3D images of biomolecules at near-atomic resolution. As such, it represents one of the most promising imaging techniques in structural biology. However, raw cryo-EM images are only highly corrupted - noisy and band-pass filtered - 2D projections o…

Accelerates MMLE using SVGD with Nesterov acceleration.

VMBPO optimizes model and policy jointly using variational lower-bound.

The detection of rare variants is important for understanding the genetic heterogeneity in mixed samples. Recently, next-generation sequencing (NGS) technologies have enabled the identification of single nucleotide variants (SNVs) in mixed samples with high resolution. Yet, the noise inherent in the biological processe…

In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expect…

We show that $k$-means (Lloyd's algorithm) is obtained as a special case when truncated variational EM approximations are applied to Gaussian Mixture Models (GMM) with isotropic Gaussians. In contrast to the standard way to relate $k$-means and GMMs, the provided derivation shows that it is not required to consider Gau…

Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structure of proteins and other macromolecular complexes at near-atomic resolution. In single particle cryo-EM, the central problem is to reconstruct the three-dimensional structure of a macromolecule from $10^{4-7}$ noisy and randomly orien…

VAEs help in learning latent variables for cryo-EM applications.

This paper tackles unsupervised speech enhancement using RVAE and proposes efficient sampling methods.

Bayesian model fuses diverse microbiome data types.

We reinterpreting the variational inference in a new perspective. Via this way, we can easily prove that EM algorithm, VAE, GAN, AAE, ALI(BiGAN) are all special cases of variational inference. The proof also reveals the loss of standard GAN is incomplete and it explains why we need to train GAN cautiously. From that, w…

Parameter estimation for model-based clustering using a finite mixture of normal inverse Gaussian (NIG) distributions is achieved through variational Bayes approximations. Univariate NIG mixtures and multivariate NIG mixtures are considered. The use of variational Bayes approximations here is a substantial departure fr…

A new diffusion model improves cryo-EM structure sampling.

A new model analyzes document structure and customer shopping patterns.

Mixture model-based clustering has become an increasingly popular data analysis technique since its introduction over fifty years ago, and is now commonly utilized within a family setting. Families of mixture models arise when the component parameters, usually the component covariance (or scale) matrices, are decompose…

Physics-informed neural networks (PINNs) [31] use automatic differentiation to solve partial differential equations (PDEs) by penalizing the PDE in the loss function at a random set of points in the domain of interest. Here, we develop a Petrov-Galerkin version of PINNs based on the nonlinear approximation of deep neur…

This paper presents a Bayesian approach to symbol and phase inference in a phase-unsynchronized digital receiver. It primarily extends [Quinn 2011] to the multi-symbol case, using the variational Bayes (VB) approximation to deal with the combinatorial complexity of the phase inference in this case. The work provides a …

This paper represents a preliminary (pre-reviewing) version of a sublinear variational algorithm for isotropic Gaussian mixture models (GMMs). Further developments of the algorithm for GMMs with diagonal covariance matrices (instead of isotropic clusters) and their corresponding benchmarking results have been published…

ProJIVE integrates multiple data types to explain joint and individual variation.

Integrates VAEs into EM for deep clustering and generation.

We introduce incremental variational inference and apply it to latent Dirichlet allocation (LDA). Incremental variational inference is inspired by incremental EM and provides an alternative to stochastic variational inference. Incremental LDA can process massive document collections, does not require to set a learning …

We introduce the variational filtering EM algorithm, a simple, general-purpose method for performing variational inference in dynamical latent variable models using information from only past and present variables, i.e. filtering. The algorithm is derived from the variational objective in the filtering setting and cons…

Given a $C^1$ planes distribution $P_T$ on all ${\mathbb R}^m$ we consider {\em horizontal $α$-harmonic maps}, $α\ge 1/2$, with respect to such a distribution. These are maps $u\in H^α({\mathbb R}^k,{\mathbb R}^m)$ satisfying $P_T\nabla u=\nabla u$ and $P_T(u)(-Δ)^αu=0$ in ${\mathcal D}'({\mathbb R}^k).$ If the distrib…

Hidden Markov Models (HMM) have been used for several years in many time series analysis or pattern recognitions tasks. HMM are often trained by means of the Baum-Welch algorithm which can be seen as a special variant of an expectation maximization (EM) algorithm. Second-order training techniques such as Variational Ba…

KLT picker automates particle picking for cryo-EM, especially for low SNR images.

We consider the second variational derivative of a given gauge-natural invariant Lagrangian taken with respect to (prolongations of) vertical parts of gauge-natural lifts of infinitesimal principal automorphisms. By requiring such a second variational derivative to vanish, {\em via} the Second Noether Theorem we find t…

Cryo-EM reconstruction is reformulated as a stochastic inverse problem to handle structural heterogeneity.

The Extreme Deconvolution method fits a probability density to a dataset where each observation has Gaussian noise added with a known sample-specific covariance, originally intended for use with astronomical datasets. The existing fitting method is batch EM, which would not normally be applied to large datasets such as…

AEVB improves understanding of latent variable models.