We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trad…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “variational approximations”
Variational methods are widely used for approximate posterior inference. However, their use is typically limited to families of distributions that enjoy particular conjugacy properties. To circumvent this limitation, we propose a family of variational approximations inspired by nonparametric kernel density estimation. …
Mixed Variational Inferencestat.ML
The Laplace approximation has been one of the workhorses of Bayesian inference. It often delivers good approximations in practice despite the fact that it does not strictly take into account where the volume of posterior density lies. Variational approaches avoid this issue by explicitly minimising the Kullback-Leibler…
We propose a family of variational approximations to Bayesian posterior distributions, called -VB, with provable statistical guarantees. The standard variational approximation is a special case of -VB with . When , a novel class of variational inequalities are developed for linking the Bayes risk …
VISA improves inference efficiency for complex models.
problem Efficient approximate inference in computationally intensive models.
method Sequential sample-average approximations within a trust region.
result VISA achieves comparable accuracy with computational savings.
Develops methods for structured variational inference with star-structured models.
problem Inference in models with interdependent variables.
method Star-structured variational inference, existence, uniqueness, self-consistency proofs, approximation error bounds, gradient-based algorithm.
result First results for existence, uniqueness, and self-consistency of variational approximations in star-structured models.
Particle-based variational inference offers a flexible way of approximating complex posterior distributions with a set of particles. In this paper we introduce a new particle-based variational inference method based on the theory of semi-discrete optimal transport. Instead of minimizing the KL divergence between the po…
A new method for efficient Gaussian process inference using sparse approximations.
problem Scalable and accurate inference for latent Gaussian processes.
method Variational approximation with sparse inverse Cholesky factors and double Kullback-Leibler minimization.
result The proposed method can achieve highly accurate approximations with polylogarithmic time complexity.
A new EVI framework improves ParVI methods by maintaining variational structure and reducing KL-divergence.
problem Improving variational inference methods for better approximation of target distributions.
method EVI framework that minimizes the VI objective function based on an energy-dissipation law, including a new 'Approximation-then-Variation' scheme.
result The new scheme significantly decreases KL-divergence and outperforms existing ParVI methods in fidelity.
Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good vari…
New framework improves variational inference for high-dimensional posteriors.
problem Challenges in choosing variational objectives and approximating families for high-dimensional posteriors.
method Conceptual framework and experimental tools to understand and optimize variational objectives and families.
result For moderate-to-high-dimensional posteriors, exclusive KL divergence is recommended due to optimization ease; for low-dimensional, heavy-tailed variational families are effective.
Boosting Variational Inference improves posterior approximations with adaptive step-sizes.
problem Limited resources hinder the widespread adoption of Boosting Variational Inference.
method Characterized global curvature impact, introduced local curvature, and developed an approximate backtracking algorithm.
result New theoretical convergence rates and experimental validation demonstrate improved performance.
Paper tightens variational GP approximations for large datasets.
problem Scaling Gaussian processes to large datasets.
method Relaxing the standard assumption about inducing points' posterior matching the prior, leading to a tighter variational approximation.
result The proposed approximation consistently matches or outperforms standard sparse variational GPs while maintaining computational cost.
The choice of approximate posterior distribution is one of the core problems in variational inference. Most applications of variational inference employ simple families of posterior approximations in order to allow for efficient inference, focusing on mean-field or other simple structured approximations. This restricti…
Inference methods are often formulated as variational approximations: these approximations allow easy evaluation of statistics by marginalization or linear response, but these estimates can be inconsistent. We show that by introducing constraints on covariance, one can ensure consistency of linear response with the var…
New method approximates diffusion process posteriors using moment functions.
problem Approximating posteriors of stochastic differential equations.
method Constructs variational process as controlled prior, approximates posterior with moment functions, uses natural gradient descent.
result Richer variational approximations for state-dependent diffusion terms.
Adaptive approximations improve variational inference for complex models.
problem Efficiently approximate marginal distributions and partition functions in complex probabilistic models.
method Two classes of adaptive approximations that include Bethe, tree-reweighted, and convex free energies.
result Proposed approximations automatically adapt to a given model and outperform existing methods.
This paper introduces a spline-based method for nonparametric ADVI that handles complex posterior distributions.
problem Learning complex posterior distributions with skewness, multimodality, and bounded support.
method Develops a spline-based nonparametric approximation approach for ADVI.
result Establishes the asymptotic consistency of the derived lower bound for importance weighted autoencoder.
We develop variational integrators from discrete Hamiltonian systems with external forces.
problem Creating accurate discrete models of continuous Hamiltonian systems.
method Constructing discrete Hamiltonian systems with external forces, analyzing symplectic structure, and combining methods to build variational integrators.
result We derive variational integrators that approximate continuous Hamiltonian systems with high accuracy.
The PAC-Bayesian approach is a powerful set of techniques to derive non- asymptotic risk bounds for random estimators. The corresponding optimal distribution of estimators, usually called the Gibbs posterior, is unfortunately intractable. One may sample from it using Markov chain Monte Carlo, but this is often too slow…
Optimized -posteriors reduce KL divergence from true posterior in parametric misspecification.
problem Reduction of KL divergence from true posterior in parametric model misspecification.
method Derivation of Bernstein-von Mises theorem and optimization of -posteriors.
result Optimized -posteriors minimize KL divergence from true posterior, especially in severe misspecification.
A new method combines Laplace and Variational Bayes for scalable inference.
problem Complex models and large datasets make exact inference infeasible.
method Low-Rank Variational Bayes Correction (VBC) using Laplace method and Variational Bayes correction in a lower dimension.
result The method ensures scalability in both model complexity and data size.
Variational inference offers scalable and flexible tools to tackle intractable Bayesian inference of modern statistical models like Bayesian neural networks and Gaussian processes. For largely over-parameterized models, however, the over-regularization property of the variational objective makes the application of vari…
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
Study variation spaces for neural networks, linking them to approximation theory.
problem Understanding the variation spaces of shallow neural networks.
method Examined variation spaces defined by convex hulls and integral representations for a dictionary of functions.
result Found that Barron space, spectral Barron space, and Radon BV space are variation spaces for certain neural networks.
Unified view of GP approximations improves efficiency.
problem Disparate variational features limit GP efficiency.
method View GP as a Banach space to unify feature selection.
result Unified understanding of existing and new features.
Improved Bayesian inference via variational approximations of generalized rho-posteriors.
problem Robust Bayesian inference under model misspecification and data contamination.
method Introducing a modified -posterior and using PAC-Bayesian analysis with variational approximations.
result Theoretical guarantees for tractable inference with competitive robustness and computational efficiency.
Bayesian model selection via mean-field variational approximation improves efficiency and accuracy.
problem Bayesian model selection under model mis-specification and latent variables.
method Mean-field variational approximation with non-asymptotic properties and geometric convergence.
result ELBO tends to select models closer to the true model than BIC as sample size increases.
Parameter estimation for model-based clustering using a finite mixture of normal inverse Gaussian (NIG) distributions is achieved through variational Bayes approximations. Univariate NIG mixtures and multivariate NIG mixtures are considered. The use of variational Bayes approximations here is a substantial departure fr…
Deep Gaussian processes provide a flexible approach to probabilistic modelling of data using either supervised or unsupervised learning. For tractable inference approximations to the marginal likelihood of the model must be made. The original approach to approximate inference in these models used variational compressio…
New method for variational inference without conjugacy constraints.
problem Efficient variational inference with flexible prior and approximation families.
method Wasserstein gradient flow for mean-field approximation.
result Improved convergence and efficiency of variational inference.
The Black Box Variational Inference (Ranganath et al. (2014)) algorithm provides a universal method for Variational Inference, but taking advantage of special properties of the approximation family or of the target can improve the convergence speed significantly. For example, if the approximation family is a transforma…
Recent advances in stochastic gradient variational inference have made it possible to perform variational Bayesian inference with posterior approximations containing auxiliary random variables. This enables us to explore a new synthesis of variational inference and Monte Carlo methods where we incorporate one or more s…
New framework improves stochastic optimization for variational inference.
problem Improving variational posterior approximations in high-dimensional models.
method Developed a robust stochastic optimization framework using Markov chains.
result Demonstrated improved accuracy and robustness across diverse models.
EigenVI uses orthogonal function expansions for efficient variational inference.
problem Efficiently approximate complex distributions in variational inference.
method EigenVI constructs variational approximations using orthogonal function expansions, minimizing Fisher divergence.
result EigenVI provides more accurate approximations than existing methods for Gaussian BBVI.
Develops a fast variational approximation for high-dimensional empirical Bayes posteriors.
problem Optimal posterior computation in high-dimensional settings with prior tails effect.
method Variational approximation of empirical Bayes posterior with data-driven centers and thin-tailed conjugate priors.
result Retains optimal concentration rate properties and superior performance compared to existing methods.
Many recent advances in large scale probabilistic inference rely on variational methods. The success of variational approaches depends on (i) formulating a flexible parametric family of distributions, and (ii) optimizing the parameters to find the member of this family that most closely approximates the exact posterior…
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
Recently, variational approximations such as the mean field approximation have received much interest. We extend the standard mean field method by using an approximating distribution that factorises into cluster potentials. This includes undirected graphs, directed acyclic graphs and junction trees. We derive generaliz…
This work improves variational inference by reducing gradient variance.
problem Hard optimization of flexible variational distributions.
method Control variate based on quadratic approximation of the model's mean and covariance.
result Significant improvement in gradient variance and optimization convergence.
Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solv…
Extending Mean-Field Variational Inference via Entropic Regularization: Theory and Computationstat.ML
New method improves approximate inference for Bayesian models.
problem Approximate inference for high-dimensional Bayesian models.
method Entropic regularization of mean-field variational inference.
result Improved recovery of true posterior dependency.
Variational inference methods for latent variable statistical models have gained popularity because they are relatively fast, can handle large data sets, and have deterministic convergence guarantees. However, in practice it is unclear whether the fixed point identified by the variational inference algorithm is a local…
New method improves variational inference for better posterior approximation.
problem Challenges in minimizing inclusive KL divergence for amortized variational inference.
method Likelihood-tempered sequential Monte Carlo samplers to estimate inclusive KL gradient.
result SMC-Wake method fits variational distributions more accurately than existing methods.
NeVI-Cut uses neural networks to efficiently propagate uncertainty without feedback.
problem Efficiently propagating uncertainty in downstream Bayesian analysis without feedback.
method NeVI-Cut combines neural networks and normalizing flows for variational inference.
result NeVI-Cut achieves significant computational gains and higher accuracy than traditional methods.
Improved neural network ensembles using Stein Variational Newton updates.
problem Lack of efficient second-order information in current ensemble methods.
method Proposes a novel approximate Bayesian inference method integrating Stein Variational Newton updates with scalable Hessian approximations.
result Significantly faster convergence and more accurate posterior distribution approximations.
VPR improves posterior uncertainty quantification by combining VI and predictive resampling.
problem Inaccurate posterior sampling with MCMC due to computational constraints.
method Variational predictive resampling (VPR) that uses VI's predictive strength and imputes future observations.
result VPR converges to the exact Bayesian posterior in a Gaussian location model and improves uncertainty quantification.
New variational method solves submodular maximum coverage problem efficiently.
problem Submodular maximum coverage problem in various applications.
method Variational optimization using Nemhauser divergence, alternating E and M steps.
result Efficient solution to submodular maximum coverage problem.