Boosting VI improves inference by greedily combining densities.
problem Improving tractability in Bayesian statistics.
method Adapting boosting techniques to VI, replacing a single density with a mixture of densities.
result Demonstrates convergence of boosting VI under relaxed smoothness assumptions.
ELBO of VAEs converges to a sum of three entropies.
problem Understanding the convergence of ELBO in VAEs.
method Analytical derivation of ELBO convergence for standard Gaussian VAEs.
result ELBO converges to a sum of three entropies at stationary points.
Bayesian model selection can be misled by ELBO under certain conditions.
problem Misleading model selection when using ELBO for Bayesian inference.
method Analysis of ELBO-based hyperparameter learning in a simple regression model.
result Bayesian model selection can prefer overfit models when ELBO is used, contrary to evidence.
Diffusion models optimize objectives similar to ELBO with Gaussian noise augmentation.
problem Optimizing diffusion models for high perceptual quality.
method Showed diffusion objectives are weighted ELBOs over noise levels, with Gaussian noise augmentation.
result Diffusion objectives equate to ELBO with Gaussian noise augmentation under monotonic weighting.
New ELBO formulation improves multimodal learning.
problem Existing ELBO models struggle with multimodal data.
method Proposes a generalized ELBO for multimodal data.
result Demonstrates improved performance over state-of-the-art models.
Generative models' ELBOs converge to entropy sums, proving for various models.
problem Proving convergence of ELBOs to entropy sums for various generative models.
method Proofs for individual models under realistic conditions.
result ELBOs of various generative models converge to entropy sums at all stationary points.
The natural gradient of ELBO vanishes in unconstrained optimization, simplifying learning.
problem The gap between evidence and ELBO has a vanishing natural gradient.
method Analyzes the Fisher-Rao gradient of ELBO and its implications for learning.
result Maximizing ELBO is equivalent to minimizing KL divergence, simplifying learning.
DLM for BNNs fails to improve over ELBO optimization.
problem Performance of DLM for Bayesian Neural Networks (BNNs).
method Direct Loss Minimization (DLM) compared to ELBO optimization.
result DLM does not significantly improve over ELBO optimization for BNNs.
ED-VAE improves VAEs by explicitly including entropy components in ELBO.
problem Limitations of traditional VAEs with ELBO in generating high-quality samples and interpreting latent spaces.
method Introduces ED-VAE, a re-formulation of ELBO that includes entropy and cross-entropy components.
result Significantly enhances model flexibility and improves interpretability and generative performance.
Develops unbiased variational inference method for better model performance.
problem Improving variational inference methods for better model performance.
method Defines an expressive variational family using a simple reparameterizable distribution and deep neural networks, directly optimizing the ELBO.
result Achieves tighter ELBO and better predictive performance than existing approaches at similar computational cost.
InfoNCE objective is equivalent to ELBO in RPM, linking to self-supervised learning.
problem Improving self-supervised learning methods by connecting them to variational inference.
method Recognizing RPM and showing InfoNCE as a simplified lower bound on MI, equal to ELBO in infinite sample limit.
result The actual InfoNCE objective is equal to the ELBO (up to a constant) in the infinite sample limit.
A new method for VAEs improves latent space disentanglement without violating probability laws.
problem Improving latent space disentanglement in VAEs without violating probability laws.
method Developed a Renyi VAE with a conditional distribution not learned, using Singular Value Decomposition for evaluation.
result Improved latent space disentanglement without violating probability laws.
Scalable Gaussian process models trained with unbiased stochastic ELBO.
problem Training large capacity Gaussian process models on huge datasets.
method Unbiased stochastic variational inference for scalable GPs.
result Accurate inference on large datasets with up to 10 million basis functions.
Analytical method approximates ELBO gradient in clutter problem.
problem Clutter problem in Bayesian networks with Gaussian likelihood.
method Reparameterization trick, local approximation of likelihood factors.
result Good accuracy and linear computational complexity compared to classical methods.
New entropy-based objective for sparse coding improves learning.
problem Sparse coding with probabilistic priors and non-Gaussian observables.
method Derive a solely entropy-based learning objective for sparse coding parameters.
result Fully analytical ELBO objective for sparse coding with non-trivial posterior approximations.
This paper analyzes VAE approximation errors in conditional exponential families.
problem Posterior collapse and approximation errors in VAEs.
method Analysis of ELBO objective and conditional exponential families.
result The ELBO optimizer pulls away from the likelihood optimizer towards a consistent subset of models.
New method improves model training on sequential data.
problem Training latent variable models with ELBO limits.
method Introducing FIVO, a family of tighter bounds.
result FIVO leads to substantial improvements over ELBO.
PVI improves SIVI by directly optimizing ELBO without parametric assumptions.
problem Intractable variational densities in SIVI methods.
method Particle Variational Inference (PVI) using empirical measures to approximate optimal mixing distributions.
result PVI directly optimizes the ELBO and performs favorably compared to other SIVI methods.
Quantized Variational Inference improves ELBO optimization with fast convergence.
problem Maximizing Evidence Lower Bound (ELBO) for variational inference.
method Optimal Voronoi Tesselation for variance-free gradients, Richardson extrapolation for asymptotic improvement.
result Quantized Variational Inference leads to fast convergence with comparable computational cost.
New method improves latent variable models without ignoring latent code.
problem Maximizing ELBO doesn't always yield good latent representations.
method Derive variational bounds on mutual information, derive rate-distortion curve, suggest new method.
result Demonstrates a family of models with identical ELBO but different characteristics.
A discrete diffusion model learns denoising, scoring, and bridging in different coordinates.
problem Understanding what a discrete diffusion model learns in different coordinate systems.
method Rigorous derivation of continuous-time Markov chain ELBO, Oracle Distance theorem, and exact coordinates for optimizer.
result The negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one.
VAEs improve representation learning by inverting the data-generating process through self-consistency.
problem VAEs struggle to invert the data-generating process, yet often succeed in representation learning.
method Studied VAEs in the limit of near-deterministic decoders, proving self-consistency and showing ELBO convergence to a regularized log-likelihood.
result VAEs can perform independent mechanism analysis (IMA), recovering true latent factors under specific conditions.
VarGrad reduces variance in ELBO gradient estimation for variational inference.
problem Improving the variance of gradient estimators in variational inference.
method VarGrad uses a new log-variance loss to estimate the ELBO gradient, achieving lower variance than the score function method.
result VarGrad offers a lower variance gradient estimator compared to other methods.
New method selects FMM components via variational Bayes.
problem Selecting the correct number of components in finite mixture models.
method Variational Bayes with mean-field approximation.
result Consistency of model selection based on ELBO maximization.
VAEs can use constant posterior variances under certain conditions, simplifying model training.
problem Learning variances in VAEs is challenging and unnecessary under certain conditions.
method Proof of non-trivial solutions with constant posterior variances, simplified ELBO formulation, and new sampling method.
result ELBO can be simplified and optimized without learning variances, improving model performance.
One-step diffusion samplers reduce sampling time and computational costs.
problem Efficient sampling from complex distributions.
method One-step diffusion, self-distillation, deterministic flow.
result Achieves competitive sample quality with fewer evaluations.
A new method for efficient inference in VAEs using Hamiltonian Monte Carlo.
problem Efficient inference in latent variable models with VAEs.
method Optimal selection of reverse kernels in Hamiltonian Monte Carlo for low-variance unbiased ELBO estimation.
result Development of a Hamiltonian Variational Auto-Encoder (HVAE) using the reparameterization trick.
Paper establishes a density formula for diffusion models, linking target density to score function.
problem Lack of theoretical foundation for optimizing DDPMs using ELBO.
method Developed a density formula for continuous-time diffusion processes, revealing the connection between target density and score function.
result The minimizer of the ELBO objective for DDPMs nearly coincides with the true objective, providing a theoretical foundation.
SMI uses mixture models to improve SVGD's performance in Bayesian inference.
problem Variance collapse in SVGD for Bayesian inference, especially with small models.
method Generalizes SVGD to Stein mixture models, optimizing an ELBO lower bound.
result SMI avoids variance collapse and accurately estimates uncertainty for small BNNs.
Proposes a new Langevin flow approach for VAEs.
problem Difficulty in constructing low variance ELBO for VAEs with large datasets.
method Integrates Langevin dynamic with quasi-symplectic integrator to improve posterior estimation.
result Shows theoretical and practical effectiveness compared to gradient flow-based methods.
New objective function reduces posterior collapse in generative models.
problem Posterior collapse in generative models with small datasets and high latent dimensions.
method Replaces ELBO's KL divergence with MMD, introduces latent clipping.
result μ-VAE outperforms ELBO and β-VAE models in quality and stability.
ELBO converges to a sum of entropies for many generative models.
problem Understanding the convergence of variational lower bounds in unsupervised learning.
method Analyzing the ELBO for a broad class of generative models, showing it equals a sum of entropies.
result The ELBO is equal to a sum of entropies at stationary points for many generative models.
New method improves generative model performance by fully conditioning variational posteriors.
problem Inaccurate inference due to partial conditioning of variational posteriors in sequential LVMs.
method Introduces fully-conditioned approximate posteriors to improve generative model performance.
result Improves generative modelling and multi-step prediction performance.
A new objective function using Jensen-Shannon divergence improves generative learning from multiple data types.
problem Learning from multiple data types efficiently and accurately.
method Proposes a novel objective function using Jensen-Shannon divergence to approximate multimodal posteriors directly.
result The mmJSD objective optimizes an ELBO and improves generative learning tasks.
A new method for Bayesian inference using discrete variables.
problem Efficient and exact computations in Bayesian variational inference.
method DIRECT approach exploiting Kronecker matrix algebra for unbiased ELBO gradient estimation.
result Direct models can compute ELBO gradients exactly and use quasi-Newton optimization methods.
Score matching fails to train VAEs robustly, revealing autoencoding loss insights.
problem Catastrophic failure of variational score matching on VAE models.
method Analysis of existing variational score matching objectives and their equivalence to autoencoding losses.
result Score matching methods fail to produce robust VAE models, predicting poor performance.
Semi-Implicit Variational Inference (SIVI) is improved with SIVI-SM using score matching.
problem Intractable densities in variational distributions hinder SIVI training.
method SIVI-SM uses score matching to handle intractable densities in a minimax formulation.
result SIVI-SM outperforms ELBO-based SIVI methods in Bayesian inference tasks.
Lower bound for VAE training objective for binary data.
problem Finding a lower bound for the ELBO of Bernoulli VAE.
method Interpretable lower bound, modified initialization, faster training architecture, PCA for latent space dimension.
result Theoretical result and improved performance of new architecture.
This paper introduces a method to estimate log-likelihood in VAE models.
problem Difficulty in comparing models trained via ELBO due to lack of log-likelihood.
method Introduces a general upper bound to approximate model evidence.
result Efficiently approximates model evidence and compares to other bounds.
New RL algorithm GDPO improves DLM reasoning efficiency.
problem Adapting RL to DLMs for efficient, unbiased likelihood estimation.
method Group Diffusion Policy Optimization (GDPO) using semi-deterministic Monte Carlo.
result GDPO outperforms existing methods on math, reasoning, and coding benchmarks.
Tighter ELBOs can harm learning, new algorithms improve performance.
problem Theoretical and empirical evidence shows that tighter ELBOs can reduce the signal-to-noise ratio of gradient estimators, hindering learning.
method Introduce three new algorithms: PIWAE, MIWAE, CIWAE, which improve over the standard IWAE.
result New algorithms can deliver improvements over IWAE, even when measured by IWAE's performance.
This paper improves inference for non-parametric Bayesian Hawkes processes.
problem Inference of non-parametric Bayesian Hawkes processes is unscalable or slow.
method Proposes a squared sparse Gaussian process for the triggering kernel and a novel variational inference schema.
result The method accelerates inference to linear time complexity and improves model selection.
Optimizing noise variance in VAEs balances reconstruction quality and prior regularisation.
problem Balancing reconstruction quality and prior regularisation in VAEs.
method Learning the noise variance in the Gaussian likelihood to balance the ELBO loss.
result Optimizing noise variance improves VAE-generated sample quality and uncertainty.
Prevents sensitive data generation in diffusion models using labeled and unlabeled data.
problem Generating sensitive data in diffusion models using unlabeled data.
method Positive-Unlabeled Diffusion Models, approximating ELBO with labeled and unlabeled data.
result Prevents the generation of sensitive data without compromising image quality.
DDVI uses diffusion models for variational inference, improving latent variable model performance.
problem Improving variational inference in latent variable models.
method Introduces diffusion-based variational posteriors trained with a regularized ELBO.
result Outperforms alternative variational posteriors on various benchmarks and a biology task.
Linear VAEs explain posterior collapse in VAEs via local maxima in log marginal likelihood.
problem Posterior collapse in VAEs where variational posterior matches prior for some latent variables.
method Analysis of linear VAEs and their relation to pPCA, proving ELBO does not introduce spurious local maxima.
result Linear VAEs have identifiable global maxima corresponding to principal component directions, explaining posterior collapse.
The paper analyzes numerical instability in variational flows and proposes a diagnostic method.
problem Numerical instability in variational flows affects sampling, density evaluation, and ELBO estimation.
method Treated variational flows as dynamical systems, used shadowing theory for theoretical guarantees, and developed a diagnostic procedure.
result Despite numerical instability, results from variational flows can be accurate enough for practical applications.
New features reduce computational cost of variational inference.
problem Efficiently compute ELBO with reduced computational cost.
method Developed features that reduce O(M3) to O(ildeNT+MT) for large M. result Unbiased ELBO estimation with reduced computational complexity.