The paper introduces validated variational inference with practical error bounds.
problem Lack of accurate post-hoc measures for variational inference.
method The paper provides rigorous bounds on variational inference error.
result The bounds are widely applicable and computationally efficient.
New method improves variational inference for hierarchical models.
problem Limited expressivity of variational distributions in Bayesian models.
method Importance weighted hierarchical variational inference.
result Superior performance in experiments compared to existing methods.
Unified variational bounds for mutual information, addressing high-dimensional challenges.
problem Estimating and optimizing Mutual Information (MI) in high dimensions is challenging.
method Unified framework of variational lower bounds parameterized by neural networks, trading off bias and variance.
result Unified bounds flexibly trade off bias and variance, improving estimation and representation learning.
We introduce the Variational Holder (VH) bound as an alternative to Variational Bayes (VB) for approximate Bayesian inference. Unlike VB which typically involves maximization of a non-convex lower bound with respect to the variational parameters, the VH bound involves minimization of a convex upper bound to the intract…
Paper improves variational inference by tightening bounds using perturbation theory.
problem Improving variational inference's bias and KL divergence approximation.
method Revisits perturbation theory to derive corrections that tighten variational bounds.
result New bounds are tighter and more mass-covering, leading to higher likelihoods.
A new upper bound for variational inference improves the efficiency of Bayesian deep learning.
problem Improving variational inference in Bayesian deep learning.
method Presented a new upper bound (EUBO) for evidence, derived from KL-divergence and log marginal likelihood, and used SGD for optimization.
result The new upper bound (EUBO) is tighter than previous methods and outperforms state-of-the-art results in Bayesian neural networks.
Improved variational inference for logistic regression and classification.
problem Intractability of Evidence Lower Bound in variational logistic regression.
method Introducing a new bound for the expectation of softplus function, applied to variational logistic regression and Gaussian process classification.
result The new bound results in a tighter posterior and faster computation compared to Monte-Carlo methods.
This work improves VAEs using MCMC methods for better variational bounds.
problem Improving the expressiveness of variational distributions in VAEs.
method Entropy-based adaptation for MALA/HMC chains to optimize tighter variational bounds.
result Higher held-out log-likelihoods and improved generative metrics.
This tutorial derives the VAE loss function under Gaussian assumptions.
problem Computational intractability of posterior distributions in Bayesian machine learning.
method Derives the variational lower bound loss function of a standard VAE.
result The Kullback-Leibler divergence has a closed form solution under Gaussian assumptions.
Paper provides variance bounds for variational inference.
problem Understanding the variance of stochastic gradient estimators in variational inference.
method Analyzes reparameterization estimators under smoothness and location-scale assumptions.
result Gives provable bounds on gradient variance, showing they are optimal under stated conditions.
Variational inference is a powerful tool for approximate inference. However, it mainly focuses on the evidence lower bound as variational objective and the development of other measures for variational inference is a promising area of research. This paper proposes a robust modification of evidence and a lower bound for…
Paper formalizes and analyzes a new bound for variational inference.
problem Lack of theoretical guarantees in variational algorithms.
method Introduces VR-IWAE bound, a generalization of IWAE.
result VR-IWAE bound leads to unbiased gradient estimators.
TVO tightens variational inference bounds for deep models.
problem Improving variational inference bounds for deep models.
method Introduces thermodynamic variational objective (TVO) connecting variational inference and thermodynamic integration.
result TVO provides tighter lower bound to log marginal likelihood than ELBO.
Paper establishes lower bounds for non-stationary kernelized bandits.
problem Optimizing functions with noisy observations in non-stationary scenarios.
method Develops algorithm-independent lower bounds for time-varying functions under total variation constraints.
result First algorithm-independent lower bounds for time-varying kernelized bandits.
Variational Prediction simplifies Bayesian inference without test time costs.
problem Bayesian inference's computational costs and posterior predictive distribution marginalization.
method Variational Prediction learns a variational approximation to the posterior predictive distribution using a variational bound.
result Directly learns a variational approximation to the posterior predictive distribution without test time marginalization costs.
This paper introduces the variational Rényi bound (VR) that extends traditional variational inference to Rényi's alpha-divergences. This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is co…
A new method simplifies variational inference for complex models.
problem Challenges in exact Bayesian inference due to intractable integrals.
method Rewriting the lower bound on model log-likelihood using a finite sample of Gaussian latent variables.
result Demonstrated effectiveness on synthetic and real-world examples.
Semi-implicit variational inference (SIVI) is introduced to expand the commonly used analytic variational distribution family, by mixing the variational parameter with a flexible distribution. This mixing distribution can assume any density function, explicit or not, as long as independent random samples can be generat…
The paper analyzes variational autoencoders for state space models with risk bounds.
problem Analyzing the risk associated with variational autoencoders for state space models.
method Backward factorization of variational distributions to analyze excess risk, providing oracle inequalities and upper bounds.
result Explicit upper bounds on variational estimation error for state space models under strong mixing assumptions.
New algorithm uses control variates to improve multi-armed bandit performance.
problem Stochastic multi-armed bandits with auxiliary reward information.
method Developed UCB-CV algorithm using control variates for mean estimation.
result UCB-CV algorithm provides tighter confidence bounds and smaller variance.
We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference(VI), based on the Variational Dropout \cite{kingma2015}. Stochastic gradient variational Bayes (SGVB) \cite{aevb} is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, …
Uniform linear bounds on volume changes in 3D hyperbolic spaces.
problem Volume variation in hyperbolic 3-manifolds.
method Uniform linear bounds proof for drilling and filling operations.
result Uniform linear bounds on volume variation proved.
Paper finds a lower bound for estimating low-rank matrices in logistic regression.
problem Estimating low-rank coefficient matrices in logistic regression.
method Derives a minimax lower bound on the risk.
result The bound depends on matrix dimensions, rank, and sample size.
Lower bound for VAE training objective for binary data.
problem Finding a lower bound for the ELBO of Bernoulli VAE.
method Interpretable lower bound, modified initialization, faster training architecture, PCA for latent space dimension.
result Theoretical result and improved performance of new architecture.
New algorithm limits regret in changing MDPs.
problem Reinforcement learning in MDPs with time-varying rewards and transitions.
method Proposed an algorithm with performance guarantees for non-stationary policies.
result First variational regret bound for general RL setting.
Paper improves variance control in importance weighted variational bounds.
problem Improving the variance of gradient estimators for IWAE.
method Develops a novel control variate that grows SNR as √K for large K.
result Empirically, the method yields superior variance reduction for generative models.
New guarantees for black-box variational inference methods.
problem Insufficient theoretical guarantees for black-box variational inference.
method Novel convergence guarantees for stochastic optimization of variational inference.
result Provable convergence of proximal and projected stochastic gradient descent for variational inference.
The paper studies curves in Riemannian manifolds using total variation flow.
problem Analyzing the evolution of curves in Riemannian manifolds using total variation.
method Defining and proving the existence of strong solutions to the flow equations, showing variational equality, and proving convergence.
result Strong solutions converge to a constant map in finite time for non-positive sectional curvature.
New insights into variational inference using Monte Carlo estimates.
problem Improving variational bounds in latent variable models.
method Analyzing properties of Monte Carlo estimates and their impact on variational gaps.
result Negative correlation reduces variational gaps, contrary to intuition.
We present a general method for deriving collapsed variational inference algo- rithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. W…
We consider a non-stationary sequential stochastic optimization problem, in which the underlying cost functions change over time under a variation budget constraint. We propose an Lp,q-variation functional to quantify the change, which yields less variation for dynamic function sequences whose changes are constrai…
Develops a new unsupervised clustering method using Variational Information Bottleneck and Gaussian Mixture Model.
problem Unsupervised clustering of unlabeled data.
method Combines Variational Information Bottleneck and Gaussian Mixture Model in a deep neural network framework.
result Derives a new bound on the cost function and provides an algorithm for efficient computation.
The total variation distance is a core statistical distance between probability measures that satisfies the metric axioms, with value always falling in [0,1]. This distance plays a fundamental role in machine learning and signal processing: It is a member of the broader class of f-divergences, and it is related to …
New method improves GP regression by relaxing variational assumption.
problem Improving variational Gaussian processes for better predictive performance.
method Relaxing the variational assumption to a more general distribution for optimization.
result New tighter evidence lower bound for GP regression.
New bounds enable training of probabilistic models for deep networks.
problem Training scalable latent variable models for deep networks.
method Introducing new variational bounds for specific output layers of neural networks.
result Analytical bounds for certain output layers allow training without re-parameterization or Monte Carlo approximations.
Mean-field variational inference is a method for approximate Bayesian posterior inference. It approximates a full posterior distribution with a factorized set of distributions by maximizing a lower bound on the marginal likelihood. This requires the ability to integrate a sum of terms in the log joint likelihood using …
New method improves inference for hierarchical models.
problem Challenges in inference for large hierarchical models.
method Locally enhanced variational bounds with subsampling.
result Better posterior approximations than baselines.
We extend the existing framework of semi-implicit variational inference (SIVI) and introduce doubly semi-implicit variational inference (DSIVI), a way to perform variational inference and learning when both the approximate posterior and the prior distribution are semi-implicit. In other words, DSIVI performs inference …
Deep Gaussian processes provide a flexible approach to probabilistic modelling of data using either supervised or unsupervised learning. For tractable inference approximations to the marginal likelihood of the model must be made. The original approach to approximate inference in these models used variational compressio…
Study introduces indecomposability for varifolds, leading to geometric consequences.
problem Understanding the structure of varifolds and their connectedness properties.
method Introducing indecomposability and related concepts for varifolds.
result Substantial geometric consequences derived from the connectedness properties of varifolds.
Recent research has made significant progress on the problem of bounding log partition functions for exponential family graphical models. Such bounds have associated dual parameters that are often used as heuristic estimates of the marginal probabilities required in inference and learning. However these variational est…
Non-negative matrix factorization (NMF) is a knowledge discovery method that is used in many fields. Variational inference and Gibbs sampling methods for it are also wellknown. However, the variational approximation error has not been clarified yet, because NMF is not statistically regular and the prior distribution us…
The paper studies the smoothness of critical points of variational integrals on Hessian spaces.
problem The study focuses on the regularity of critical points of variational integrals defined on Hessian spaces.
method The approach involves solving a fourth order nonlinear equation and analyzing the Hessian of the critical points.
result Smooth critical points with bounded Hessian are shown to be smooth provided their Hessian has small BMO.
New algorithms solve stochastic variational inequalities without bounded variance assumption.
problem Solving stochastic variational inequalities without bounded variance assumption.
method Developed algorithms for two classes of problems: monotone and structured nonmonotone VIs.
result Oracle complexity of O(ε^-4) for solving VIs with unbounded domains and possibly unbounded variance.
This paper compares gradient estimators in importance-weighted VI and justifies the superiority of DREP over REP.
problem Understanding the impact of gradient estimators on importance-weighted VI algorithms.
method Unified theoretical comparison of reparameterized and doubly-reparameterized gradient estimators tied to IWAE, VR, and VR-IWAE bounds.
result Formally justifies the superiority of doubly-reparameterized gradient estimators over reparameterized ones in importance-weighted VI.
Black box variational inference (BBVI) with reparameterization gradients triggered the exploration of divergence measures other than the Kullback-Leibler (KL) divergence, such as alpha divergences. In this paper, we view BBVI with generalized divergences as a form of estimating the marginal likelihood via biased import…
Paper improves variational inference on Boolean hypercube using quantum methods.
problem Improving variational inference for pairwise Markov random fields on the Boolean hypercube.
method Quantum relaxations of the Kullback-Leibler divergence for upper-bounds, primal-dual optimization, and greedy selection of hierarchies.
result Efficient algorithm and improved bounds for variational inference.
Quantized Variational Inference improves ELBO optimization with fast convergence.
problem Maximizing Evidence Lower Bound (ELBO) for variational inference.
method Optimal Voronoi Tesselation for variance-free gradients, Richardson extrapolation for asymptotic improvement.
result Quantized Variational Inference leads to fast convergence with comparable computational cost.