Optimized α-posteriors reduce KL divergence from true posterior in parametric misspecification.
problem Reduction of KL divergence from true posterior in parametric model misspecification.
method Derivation of Bernstein-von Mises theorem and optimization of α-posteriors. result Optimized α-posteriors minimize KL divergence from true posterior, especially in severe misspecification. Variational Prediction simplifies Bayesian inference without test time costs.
problem Bayesian inference's computational costs and posterior predictive distribution marginalization.
method Variational Prediction learns a variational approximation to the posterior predictive distribution using a variational bound.
result Directly learns a variational approximation to the posterior predictive distribution without test time marginalization costs.
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
Robust VB framework for large datasets with outliers.
problem Handling outliers and contamination in large datasets.
method Divide and conquer approach with geometric median aggregation.
result VM-Posterior distribution preserves contraction properties.
New methods for scalable inference in modular models with misspecified sub-models.
problem Model misspecification in multi-modular models complicates evidence combination.
method Variational methods for approximating Cut and SMI posteriors, and Variational Meta-Posterior.
result Feasibility of analysis with multiple cuts using a single set of variational parameters.
Adaptive variational Bayes framework improves inference adaptively.
problem Lack of general and computationally tractable variational Bayes method for adaptive inference.
method Proposes a novel adaptive variational Bayes framework combining variational posteriors over individual models.
result Adaptive variational Bayes achieves optimal contraction rates adaptively under general conditions.
Variational language models seek to estimate the posterior of latent variables with an approximated variational posterior. The model often assumes the variational posterior to be factorized even when the true posterior is not. The learned variational posterior under this assumption does not capture the dependency relat…
PVI seeks a posterior that makes predictions closer to true data, not approximating the Bayesian posterior.
problem Finding meaningful posterior distributions under model misspecification.
method Predictive variational inference (PVI) seeks an optimal posterior density for close predictive matching to true data.
result PVI learns a posterior that is not the same as the Bayesian posterior, but is closer to the true data generating process.
Proposes MIVI for efficient posterior estimation and design of MCMC transitions.
problem Efficiently estimating posterior distributions in constrained time.
method Combines variational inference and MCMC with a variational distribution and optimized Markov chain.
result Optimized Markov chain improves variational distribution and vice versa, leading to more accurate posteriors.
The Laplace approximation has been one of the workhorses of Bayesian inference. It often delivers good approximations in practice despite the fact that it does not strictly take into account where the volume of posterior density lies. Variational approaches avoid this issue by explicitly minimising the Kullback-Leibler…
Improved Bayesian inference via variational approximations of generalized rho-posteriors.
problem Robust Bayesian inference under model misspecification and data contamination.
method Introducing a modified ρ-posterior and using PAC-Bayesian analysis with variational approximations. result Theoretical guarantees for tractable inference with competitive robustness and computational efficiency.
VPR improves posterior uncertainty quantification by combining VI and predictive resampling.
problem Inaccurate posterior sampling with MCMC due to computational constraints.
method Variational predictive resampling (VPR) that uses VI's predictive strength and imputes future observations.
result VPR converges to the exact Bayesian posterior in a Gaussian location model and improves uncertainty quantification.
Variational autoencoders often collapse, showing latent variables are non-identifiable.
problem Posterior collapse in variational autoencoders due to non-identifiable latent variables.
method Proves latent variable non-identifiability causes posterior collapse. Proposes latent-identifiable models using Brenier maps and input convex neural networks.
result Latent-identifiable models resolve posterior collapse and provide meaningful representations.
Geometric framework analyzes bias in variational inference for posterior functionals.
problem Analyzing the bias of posterior functionals under variational approximations.
method Developed a geometric framework to evaluate the bias of posterior functionals using the variational tangent space.
result The leading-order bias of a posterior functional is determined by its component orthogonal to the variational tangent space.
We study convergence rates of variational posterior distributions for nonparametric and high-dimensional inference. We formulate general conditions on prior, likelihood, and variational class that characterize the convergence rates. Under similar "prior mass and testing" conditions considered in the literature, the rat…
Improved VAE estimation from incomplete data using variational mixtures.
problem Estimating VAEs from incomplete data increases posterior complexity.
method Introducing variational mixtures based on finite and imputation distributions.
result Variational mixtures improve VAE estimation accuracy from incomplete data.
Variational inference struggles with weight symmetries in neural networks, leading to biased posteriors.
problem Weight space symmetries in neural networks cause multimodal posteriors, challenging variational inference.
method Developed a symmetrization mechanism to create permutation invariant variational posteriors.
result Symmetrized variational posteriors have a better fit to the true posterior and improved predictive performance.
A new method learns posterior and predictive distributions together, reducing computational cost.
problem Sequential two-stage Bayesian inference is computationally expensive.
method Amortized variational inference targeting posterior-predictive distribution.
result Efficient online inference with more accurate predictive distributions.
The choice of approximate posterior distribution is one of the core problems in variational inference. Most applications of variational inference employ simple families of posterior approximations in order to allow for efficient inference, focusing on mean-field or other simple structured approximations. This restricti…
Develops a fast variational approximation for high-dimensional empirical Bayes posteriors.
problem Optimal posterior computation in high-dimensional settings with prior tails effect.
method Variational approximation of empirical Bayes posterior with data-driven centers and thin-tailed conjugate priors.
result Retains optimal concentration rate properties and superior performance compared to existing methods.
New framework improves variational inference for high-dimensional posteriors.
problem Challenges in choosing variational objectives and approximating families for high-dimensional posteriors.
method Conceptual framework and experimental tools to understand and optimize variational objectives and families.
result For moderate-to-high-dimensional posteriors, exclusive KL divergence is recommended due to optimization ease; for low-dimensional, heavy-tailed variational families are effective.
DDVI uses diffusion models for variational inference, improving latent variable model performance.
problem Improving variational inference in latent variable models.
method Introduces diffusion-based variational posteriors trained with a regularized ELBO.
result Outperforms alternative variational posteriors on various benchmarks and a biology task.
Variational auto-encoders (VAE) are scalable and powerful generative models. However, the choice of the variational posterior determines tractability and flexibility of the VAE. Commonly, latent variables are modeled using the normal distribution with a diagonal covariance matrix. This results in computational efficien…
Sparse variational approximations allow for principled and scalable inference in Gaussian Process (GP) models. In settings where several GPs are part of the generative model, theses GPs are a posteriori coupled. For many applications such as regression where predictive accuracy is the quantity of interest, this couplin…
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive poste…
We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trad…
BF-VI improves posterior approximation in complex models.
problem Inefficient posterior approximations in complex models.
method Combines normalizing flows and Bernstein polynomial transformations.
result BF-VI outperforms other VI methods in approximating complex multivariate posteriors.
New method improves generative model performance by fully conditioning variational posteriors.
problem Inaccurate inference due to partial conditioning of variational posteriors in sequential LVMs.
method Introduces fully-conditioned approximate posteriors to improve generative model performance.
result Improves generative modelling and multi-step prediction performance.
SoftCVI uses contrastive estimation to infer complex posteriors.
problem Estimating complex posteriors in Bayesian inference.
method Contrastive variational inference with self-generated soft labels.
result SoftCVI outperforms other variational approaches in stability and coverage.
FTIP uses normalizing flows to improve posterior inference in function space.
problem Challenges in posterior inference with implicit-process priors.
method FTIP uses normalizing flows to define a richer variational distribution over combination weights.
result FTIP captures asymmetric and multimodal posterior structure better than Gaussian coefficient approximations.
Variational inference methods often focus on the problem of efficient model optimization, with little emphasis on the choice of the approximating posterior. In this paper, we review and implement the various methods that enable us to develop a rich family of approximating posteriors. We show that one particular method …
Recent progress in variational inference has paid much attention to the flexibility of variational posteriors. One promising direction is to use implicit distributions, i.e., distributions without tractable densities as the variational posterior. However, existing methods on implicit posteriors still face challenges of…
Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
This work tackles posterior collapse in conditional and hierarchical VAEs.
problem Posterior collapse in VAEs leads to poor latent variable representations.
method Theoretical analysis of linear conditional and hierarchical VAEs, empirical validation.
result Theoretical and empirical evidence of posterior collapse causes in conditional and hierarchical VAEs.
Study trade-offs between statistical and computational efficiency in variational inference.
problem Optimizing statistical accuracy vs. computational efficiency in Bayesian inference.
method Case study on Gaussian inferential models with diagonal plus low-rank precision matrices, analyzing Bayesian posterior inference and frequentist uncertainty quantification errors.
result Lower-rank models reduce variance and accelerate convergence but increase posterior inference error.
A scalable method for efficient inference in Gaussian process regression networks.
problem Intractable inference in Gaussian process regression networks (GPRN).
method Tensorization of output space, tensor/matrix-normal variational posteriors, joint optimization, and exploiting Kronecker product structure.
result Captures posterior dependencies and improves inference quality for large number of outputs.
Empirical Bayes rates via variational approximations and prior decomposition.
problem Nonparametric and high-dimensional inference convergence rates.
method Variational perspective and prior decomposition.
result Empirical Bayes posterior rates derived from variational Bayes.
We propose a simple and general variant of the standard reparameterized gradient estimator for the variational evidence lower bound. Specifically, we remove a part of the total derivative with respect to the variational parameters that corresponds to the score function. Removing this term produces an unbiased gradient …
Paper establishes statistical validity for variational Bayes in neural networks.
problem Lack of theoretical validity for Variational Bayes in Bayesian Neural Networks.
method Establishes posterior consistency for mean-field variational posterior in feed-forward neural networks.
result Proves VP concentrates around Hellinger neighborhoods of true density function under certain conditions.
This paper introduces a spline-based method for nonparametric ADVI that handles complex posterior distributions.
problem Learning complex posterior distributions with skewness, multimodality, and bounded support.
method Develops a spline-based nonparametric approximation approach for ADVI.
result Establishes the asymptotic consistency of the derived lower bound for importance weighted autoencoder.
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a …
Variational inference (VI) provides fast approximations of a Bayesian posterior in part because it formulates posterior approximation as an optimization problem: to find the closest distribution to the exact posterior over some family of distributions. For practical reasons, the family of distributions in VI is usually…
Improves decision-making in models fit with AEVB by using distinct approximate posteriors.
problem Bias in expected risk estimates due to variational distribution use.
method Use multiple approximate posteriors, including those distinct from variational, for decision-making.
result Proposed approach outperforms state-of-the-art methods in single-cell RNA sequencing.
New method DDVI improves posterior inference for deep Gaussian processes.
problem Inference of inducing points in DGPs is challenging and biased.
method DDVI uses denoising diffusion SDE and score matching for posterior approximation.
result Empirically shows DDVI outperforms baseline methods in inducing point inference.
New method approximates diffusion process posteriors using moment functions.
problem Approximating posteriors of stochastic differential equations.
method Constructs variational process as controlled prior, approximates posterior with moment functions, uses natural gradient descent.
result Richer variational approximations for state-dependent diffusion terms.
Particle-based variational inference offers a flexible way of approximating complex posterior distributions with a set of particles. In this paper we introduce a new particle-based variational inference method based on the theory of semi-discrete optimal transport. Instead of minimizing the KL divergence between the po…
Learning latent variable models with stochastic variational inference is challenging when the approximate posterior is far from the true posterior, due to high variance in the gradient estimates. We propose a novel rejection sampling step that discards samples from the variational posterior which are assigned low likel…