The paper analyzes mean-field variational Bayes for complex models and proposes new uncertainty quantification methods.
problem Approximating posterior distributions in complex Bayesian models with latent variables.
method Non-asymptotic analysis on mean-field variational inference, showing that a normal distribution with the MLE center approximates the posterior well.
result The mean-field approximation matches the MLE up to higher-order terms and is essentially efficient for regular parametric models.
New method for high-dimensional linear regression using empirical Bayes.
problem Estimating prior in high-dimensional linear regression.
method Variational empirical Bayes approach with NPMLE and mean field approximation.
result Established asymptotic consistency and computational efficiency of the method.
Efficiently identifies important variables in binary outcomes using variational Bayes.
problem Bayesian variable selection for binary outcomes with computational challenges.
method Mean-field variational Bayes approximation with closed-form updates and efficient inference algorithm.
result Successfully identifies important variables and is orders of magnitude faster than MCMC.
The mean field variational Bayes method is becoming increasingly popular in statistics and machine learning. Its iterative Coordinate Ascent Variational Inference algorithm has been widely applied to large scale Bayesian inference. See Blei et al. (2017) for a recent comprehensive review. Despite the popularity of the …
A scalable method for accurate inference of low-dimensional parameters in high-dimensional linear regression.
problem Statistical inference for low-dimensional parameters in high-dimensional linear regression models.
method Mean-field variational Bayes approach, focusing on nuisance parameters and conditional distributions.
result Competitive numerical performance and theoretical guarantees for estimation and uncertainty quantification.
Mean Field Variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is its (sometimes severe) underestimates of the uncertainty of model variables and lack of information about model variable covariance.…
Improved Bayesian uncertainty quantification using variational bagging.
problem Inefficient and underestimating uncertainty in mean-field variational Bayes.
method Integrates bagging with variational Bayes for improved inference.
result Bagged variational posterior provides proper uncertainty quantification.
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable cova…
New scalable variational Bayes methods for Hawkes processes.
problem Computational intractability of Bayesian estimation for generalised nonlinear Hawkes processes.
method Unified variational Bayes framework, adaptive mean-field approximation, sparsity-inducing procedure.
result Adaptive mean-field variational algorithm for sigmoid Hawkes processes is scalable and robust.
We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse tr…
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable cova…
A mean field variational Bayes approach to support vector machines (SVMs) using the latent variable representation on Polson & Scott (2012) is presented. This representation allows circumvention of many of the shortcomings associated with classical SVMs including automatic penalty parameter selection, the ability to ha…
The paper studies stability of mean-field variational inference for log-concave distributions.
problem Stability of mean-field variational inference for log-concave distributions.
method Novel approach via linearized optimal transport, lifting non-convex problem to convex optimization over transport maps.
result Dimension-free Lipschitz continuity of the MFVI optimizer with respect to the target distribution, measured in 2-Wasserstein distance.
The article addresses a long-standing open problem on the justification of using variational Bayes methods for parameter estimation. We provide general conditions for obtaining optimal risk bounds for point estimates acquired from mean-field variational Bayesian inference. The conditions pertain to the existence of cer…
Improved VB algorithm for high-dimensional logistic regression with theoretical guarantees.
problem Sparse high-dimensional logistic regression model selection.
method Spike and slab variational Bayes approximation.
result Optimal convergence rates in ℓ2 and prediction loss for sparse truths. Flexible Bayesian approach for generalized linear models, especially for sparse logistic regression.
problem Sparse logistic regression challenges in machine learning.
method Empirical Bayes approach with mean-field variational inference, tuning-free and scalable.
result Superior predictive performance in sparse logistic regression compared to existing methods.
The paper offers streamlined algorithms for fitting complex linear mixed models.
problem Linear mixed models with crossed random effects in large dimensions.
method Mean field variational Bayes algorithms with various relaxations and storage strategies.
result Different inference strategies have varying trade-offs between accuracy and computational demands.
Clarifies EM algorithm and variational Bayesian inference concepts.
problem Gaps in AI literature understanding of EM and variational concepts.
method Tutorial presentation of EM algorithm, variational Bayesian inference, and autoencoded variational Bayes.
result Establishes clear links between EM and variational methods.
Bayesian models provide a framework for probabilistic modelling of complex datasets. However, many of such models are computationally demanding especially in the presence of large datasets. On the other hand, in sensor network applications, statistical (Bayesian) parameter estimation usually needs distributed algorithm…
The paper analyzes high-dimensional linear regression using parametric empirical Bayes methods.
problem Estimation of i.i.d. priors in high-dimensional Bayesian linear regression with random design.
method Parametric empirical Bayes estimation, variational lower bound maximization, phase transition analysis.
result The vEB estimator is information theoretically optimal up to p=o(n2/3) but sub-optimal in higher dimensions. Paper derives CLT for Bayesian neural networks trained with variational inference.
problem Analyzing the fluctuation behavior of Bayesian neural networks trained with different variational inference schemes.
method Rigorous derivation of CLT for three variational inference schemes: idealized, Bayes-by-Backprop, and Minimal VI.
result Minimal VI scheme has larger variances but is more computationally efficient.
We propose a family of variational approximations to Bayesian posterior distributions, called α-VB, with provable statistical guarantees. The standard variational approximation is a special case of α-VB with α=1. When α∈(0,1], a novel class of variational inequalities are developed for linking the Bayes risk …
New approach to meta-learning with variational Bayes for unlabeled data.
problem Improving machine learning systems' adaptability to small data.
method Generative meta-learning using AEVB and mean-field variational Bayes.
result Relevant VB updates do not depend on generative neural networks for certain models.
New method selects FMM components via variational Bayes.
problem Selecting the correct number of components in finite mixture models.
method Variational Bayes with mean-field approximation.
result Consistency of model selection based on ELBO maximization.
Robust VB framework for large datasets with outliers.
problem Handling outliers and contamination in large datasets.
method Divide and conquer approach with geometric median aggregation.
result VM-Posterior distribution preserves contraction properties.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Bayesian entity resolution merges together multiple, noisy databases and returns the minimal collection of unique individuals represented, together with their true, latent record values. Bayesian methods allow flexible generative models that share power across databases as well as principled quantification of uncertain…
Variational Bayes (VB), also known as independent mean-field approximation, has become a popular method for Bayesian network inference in recent years. Its application is vast, e.g. in neural network, compressed sensing, clustering, etc. to name just a few. In this paper, the independence constraint in VB will be relax…
Explaining how overparametrized neural networks simultaneously achieve low risk and zero empirical risk on benchmark datasets is an open problem. PAC-Bayes bounds optimized using variational inference (VI) have been recently proposed as a promising direction in obtaining non-vacuous bounds. We show empirically that thi…
This work introduces a fixed-point optimization for variational inference.
problem Improving quantified uncertainty in predictions by optimizing a simplified distribution over parameters.
method Projective integral updates for high-dimensional variational inference.
result Efficient quasirandom quadrature sequence for mean-field distributions, leading to quasi-Newton variational Bayes (QNVB).
Paper establishes statistical validity for variational Bayes in neural networks.
problem Lack of theoretical validity for Variational Bayes in Bayesian Neural Networks.
method Establishes posterior consistency for mean-field variational posterior in feed-forward neural networks.
result Proves VP concentrates around Hellinger neighborhoods of true density function under certain conditions.
DADVI improves ADVI by using deterministic approximation for faster, more accurate posterior estimation.
problem Intractable posterior uncertainty estimates and lack of clear convergence criteria in ADVI.
method Replaces stochastic MFVB objective with deterministic Monte Carlo approximation (SAA) and uses second-order optimization.
result DADVI provides faster and more accurate posterior estimates with default settings.
Proposes VAE-KRnet for density estimation and variational Bayes.
problem Challenges of density estimation and posterior approximation in high dimensions.
method Combines VAE and KRnet for density estimation, incorporating mutual information maximization.
result Improves posterior approximation in high-dimensional cases.
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
A new particle algorithm improves mean-field variational inference.
problem Efficiently approximating nonparametric posterior distributions in machine learning.
method Introduces PArticle VI (PAVI), a novel particle-based algorithm for nonparametric mean-field approximation.
result Obtains non-asymptotic error bounds for PArticle VI, providing the first end-to-end guarantee for particle-based MFVI.
A new model predicts multivariate regression using similarities to data points.
problem Complex, high-dimensional input-output relationships.
method Bayesian mixture-of-experts with conditional Gaussian mixtures and variational Bayes.
result Outperforms competitors in high-dimensional settings.
HEBAE improves VAEs by adaptively balancing reconstruction and regularization.
problem Posterior collapse in VAEs leading to over-regularization and poor latent encoding.
method Hierarchical Empirical Bayes approach to probabilistic generative models.
result HEBAE generates higher quality samples with better FID scores.
Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. The result is a rich and meaningful prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the …
Empirical Bayes rates via variational approximations and prior decomposition.
problem Nonparametric and high-dimensional inference convergence rates.
method Variational perspective and prior decomposition.
result Empirical Bayes posterior rates derived from variational Bayes.
Adaptive variational Bayes framework improves inference adaptively.
problem Lack of general and computationally tractable variational Bayes method for adaptive inference.
method Proposes a novel adaptive variational Bayes framework combining variational posteriors over individual models.
result Adaptive variational Bayes achieves optimal contraction rates adaptively under general conditions.
Recently, variational approximations such as the mean field approximation have received much interest. We extend the standard mean field method by using an approximating distribution that factorises into cluster potentials. This includes undirected graphs, directed acyclic graphs and junction trees. We derive generaliz…
VBphenoR uses variational Bayes for EHR-based patient phenotyping.
problem Phenotyping patients from EHR data for targeted treatments.
method Variational Bayes Gaussian Mixture Model (GMM) and logistic regression.
result Closed-form inference for efficient patient phenotype determination.
The digital telecommunications receiver is an important context for inference methodology, the key objective being to minimize the expected loss function in recovering the transmitted information. For that criterion, the optimal decision is the Bayesian minimum-risk estimator. However, the computational load of the Bay…
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
Simplified Variational Bayes for easier inference.
problem Complex derivation of Variational Bayes.
method 3-step recipe to identify posterior form and directly write updates.
result Easier, faster, shorter derivation of Variational Bayes.
A new method combines Laplace and Variational Bayes for scalable inference.
problem Complex models and large datasets make exact inference infeasible.
method Low-Rank Variational Bayes Correction (VBC) using Laplace method and Variational Bayes correction in a lower dimension.
result The method ensures scalability in both model complexity and data size.
Unified empirical and variational Bayes for unnormalized densities.
problem Approximating unnormalized densities using latent variable models.
method Formulate a latent variable model for Y=X+N(0,σ2Id), use ELBO as parametrization of Y's energy function, and estimate X with empirical Bayes least-squares. result UVB has higher capacity to approximate energy functions than MLPs in DEEN.
A practical guide to Variational Bayes methods.
problem Bayesian inference for data analysis practitioners.
method Explains and demonstrates commonly used Variational Bayes methods.
result Quick derivation and implementation of VB algorithms.