A new particle algorithm improves mean-field variational inference.
problem Efficiently approximating nonparametric posterior distributions in machine learning.
method Introduces PArticle VI (PAVI), a novel particle-based algorithm for nonparametric mean-field approximation.
result Obtains non-asymptotic error bounds for PArticle VI, providing the first end-to-end guarantee for particle-based MFVI.
New method accelerates energetic variational inference using particle dynamics.
problem Efficiently solving variational inference problems with reduced computational cost.
method Particle-based variational inference with implicit scheme, inspired by energy quadratization and operator splitting.
result Significantly reduces computational cost compared to existing methods.
A new EVI framework improves ParVI methods by maintaining variational structure and reducing KL-divergence.
problem Improving variational inference methods for better approximation of target distributions.
method EVI framework that minimizes the VI objective function based on an energy-dissipation law, including a new 'Approximation-then-Variation' scheme.
result The new scheme significantly decreases KL-divergence and outperforms existing ParVI methods in fidelity.
Particle-based variational inference offers a flexible way of approximating complex posterior distributions with a set of particles. In this paper we introduce a new particle-based variational inference method based on the theory of semi-discrete optimal transport. Instead of minimizing the KL divergence between the po…
Paper formulates particle flow using variational inference and Fisher-Rao gradient flow.
problem Estimating posterior densities in probabilistic models.
method Variational formulation of particle flow, Fisher-Rao gradient flow, Gaussian and Gaussian mixture approximations.
result Gaussian and Gaussian mixture approximations of Fisher-Rao particle flow reduce to Exact Daum and Huang particle flow under linear Gaussian assumptions.
MPM-ParVI uses particle sampling for variational inference.
problem Variational inference for complex probabilistic models.
method Material Point Method (MPM) for particle-based simulation.
result Deterministic sampling and inference for intractable densities.
A new variational method for SSMs improves inference efficiency.
problem Hard variational inference for state space models.
method Proposes variational marginal particle filter (VMPF) based on Rao-Blackwellization.
result VMPF provides tighter variational bounds and sometimes benefits from unbiased reparameterization.
SPH-ParVI uses fluid dynamics to sample unknown densities efficiently.
problem Sampling partially known densities or using gradients in probabilistic models.
method Smoothed Particle Hydrodynamics (SPH) for modeling fluid dynamics to approximate target densities.
result SPH-ParVI provides fast, flexible, scalable, and deterministic sampling for Bayesian inference and generative models.
A new ParVI framework improves particle-based variational inference methods.
problem Non-trivial kernel design in particle-based variational inference methods.
method Proposes a generalized Wasserstein gradient descent (GWG) framework with broader regularizers.
result Demonstrates strong convergence guarantees and effectiveness on simulated and real data.
EnKO combines VI and EnKF for efficient latent dynamics inference.
problem Particle degeneracy and biased gradient estimators in SMC-based methods.
method EnKO: hybrid of VI and EnKF.
result EnKO outperforms SMC-based methods in predictive ability and particle efficiency.
Paper explores SVGD for Bayesian inference, linking deterministic and stochastic dynamics.
problem Bayesian inference and Markov chain Monte Carlo methods.
method Stein variational gradient descent (SVGD) with deterministic and stochastic dynamics.
result Identifies Stein-Fisher information as the leading order contribution in the long-time and many-particle regime.
SMI uses mixture models to improve SVGD's performance in Bayesian inference.
problem Variance collapse in SVGD for Bayesian inference, especially with small models.
method Generalizes SVGD to Stein mixture models, optimizing an ELBO lower bound.
result SMI avoids variance collapse and accurately estimates uncertainty for small BNNs.
PVI improves SIVI by directly optimizing ELBO without parametric assumptions.
problem Intractable variational densities in SIVI methods.
method Particle Variational Inference (PVI) using empirical measures to approximate optimal mixing distributions.
result PVI directly optimizes the ELBO and performs favorably compared to other SIVI methods.
Differentiable PF via entropy-regularized OT for better inference.
problem Non-differentiability of traditional PF resampling methods.
method Entropy-regularized optimal transport for differentiable resampling.
result Convergent differentiable PF method with improved gradient estimates.
Stein variational gradient descent (SVGD) is a recently proposed particle-based Bayesian inference method, which has attracted a lot of interest due to its remarkable approximation ability and particle efficiency compared to traditional variational inference and Markov Chain Monte Carlo methods. However, we observed th…
Improves SVGD for high-dimensional Bayesian inference by reducing variance collapse.
problem Variance collapse in SVGD reduces accuracy and diversity of estimation.
method Augmented Message Passing SVGD (AUMP-SVGD) method, a two-stage optimization procedure.
result AUMP-SVGD achieves satisfactory accuracy and overcomes variance collapse in various benchmark problems.
A new algorithm for optimizing probability distributions converges linearly.
problem Optimizing functionals over families of probability distributions.
method Variational transport: particle-based algorithm approximating Wasserstein gradient descent.
result Variational transport converges linearly to the global minimum of the objective functional.
New particle-based VI algorithm expands function class and improves scalability.
problem Limited function class in particle-based VI algorithms restricts flexibility and scalability.
method Introduces a functional regularization term to expand the function class and proposes PFG algorithm.
result Proposed PFG algorithm has larger function class, improved scalability, better adaptation to ill-conditioned distributions, and provable convergence.
New coin sampling method for Bayesian inference without learning rates.
problem Scalable Bayesian inference with learning rate tuning issues.
method Coin sampling for gradient-based Bayesian inference.
result Comparable performance to other ParVI algorithms without learning rate tuning.
Generative ParVI learns flexible sampling from posterior distributions.
problem Avoiding arbitrary parametric assumptions in variational inference.
method Neural sampler trained with functional gradient of KL-divergence.
result GPVI outperforms previous generative ParVI methods and is competitive with other approaches.
BSVGD improves sampling for multimodal distributions using branching.
problem Sampling from multimodal distributions.
method Random branching in Stein Variational Gradient Descent (SVGD).
result Theoretical convergence guarantee and empirical validation.
Particle-based variational inference methods (ParVIs) have gained attention in the Bayesian inference literature, for their capacity to yield flexible and accurate approximations. We explore ParVIs from the perspective of Wasserstein gradient flows, and make both theoretical and practical contributions. We unify variou…
Paper proposes LSVGD to stabilize GAN training via Langevin Stein Variational Gradient Descent.
problem Mode collapse and performance deterioration in GAN training.
method Langevin Stein Variational Gradient Descent (LSVGD) incorporating noise to stabilize training.
result LSVGD improves performance and stability of various GAN models.
New samplers minimize KL divergence for constrained and non-Euclidean geometries.
problem Efficient sampling from constrained and non-Euclidean distributions.
method Stein Variational Mirror Descent and Mirrored Stein Variational Gradient Descent.
result New samplers converge more rapidly and accurately than prior methods.
The paper connects neural network ensembles to Bayesian inference using variational methods.
problem Explaining the behavior of ensemble methods in neural networks.
method Deriving conditions for ensemble optimization to reduce divergence to the posterior distribution.
result Ensemble methods can be a valid alternative to approximate Bayesian inference.
Online VSMC efficiently learns SSM parameters in streaming data.
problem Parameter learning and latent state inference in state-space models.
method Combines particle methods and variational inference for online learning.
result Online VSMC achieves efficient, entirely on-the-fly parameter estimation and particle proposal adaptation.
SIFG uses noisy particles to efficiently sample from complex distributions.
problem Efficient sampling from complex distributions using particle-based methods.
method SIFG introduces a semi-implicit functional gradient flow with Gaussian noise to improve sampling efficiency and accuracy.
result SIFG achieves strong theoretical convergence guarantees and efficient sampling.
Gaussian-SVGD dynamics converge to Gaussian distributions under certain conditions.
problem Understanding the theoretical properties of SVGD, especially for Gaussian targets.
method Detailed theoretical study of Gaussian-SVGD dynamics, considering both mean-field PDE and discrete particle systems.
result Gaussian-SVGD dynamics converge linearly to the Gaussian distribution closest to the target in KL divergence.
DSVGD improves federated learning with fewer communication rounds.
problem Federated learning scalability and trustworthiness.
method Distributed Stein Variational Gradient Descent (DSVGD) for non-parametric Bayesian inference.
result DSVGD achieves comparable accuracy and scalability to other methods, with well-calibrated predictions.
DriftLite improves inference quality of diffusion models without retraining.
problem Adapting pre-trained diffusion models to new target distributions without retraining.
method Lightweight, training-free particle-based approach that steers inference dynamics with optimal stability control.
result Consistently reduces variance and improves sample quality over existing methods.
New algorithm trains latent diffusion models using interacting particles.
problem Training latent diffusion models efficiently and accurately.
method Reformulate training as minimizing a free energy functional, then approximate with interacting particles.
result The new algorithm outperforms previous methods in experiments.
New method for efficient Bayesian inference in GPSSMs.
problem Challenges in inference for Gaussian process state-space models.
method Free-form variational inference with stochastic gradient Hamiltonian Monte Carlo.
result Our method learns transition dynamics and latent states more accurately than competing methods.
Paper improves particle variational inference by optimizing generalization error bound.
problem Improving the diversity of models in particle variational inference to enhance generalization.
method Develops a new second-order Jensen inequality with a repulsion term based on the loss function, leading to a tighter generalization error bound.
result The proposed PVI optimizes the generalization error bound directly, improving performance compared to existing methods.
Approximate inference in high-dimensional, discrete probabilistic models is a central problem in computational statistics and machine learning. This paper describes discrete particle variational inference (DPVI), a new approach that combines key strengths of Monte Carlo, variational and search-based techniques. DPVI is…
This study analyzes theoretical guarantees for VI with fixed-variance Gaussian mixtures.
problem Theoretical analysis of variational inference with non-Gaussian distributions.
method Formulates variational inference as minimizing a mollified relative entropy, solving it through gradient descent on particle positions.
result Establishes descent lemma and approximation error bounds for optimization of variational inference.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
New method for constrained sampling using gradient flows.
problem Sampling from constrained domains.
method Introducing a boundary condition for gradient flow to confine particles within the domain.
result Provable continuous-time convergence in total variation for constrained sampling.
While Bayesian neural networks (BNNs) have drawn increasing attention, their posterior inference remains challenging, due to the high-dimensional and over-parameterized nature. To address this issue, several highly flexible and scalable variational inference procedures based on the idea of particle optimization have be…
Proposes PGPS for efficient Bayesian inference.
problem Efficient sampling from complex posterior distributions.
method Path-guided particle-based sampling with Log-weighted Shrinkage.
result PGPS generates samples closer to target distribution.
We propose in this work RBM-SVGD, a stochastic version of Stein Variational Gradient Descent (SVGD) method for efficiently sampling from a given probability measure and thus useful for Bayesian inference. The method is to apply the Random Batch Method (RBM) for interacting particle systems proposed by Jin et al to the …
We present a scalable approach to performing approximate fully Bayesian inference in generic state space models. The proposed method is an alternative to particle MCMC that provides fully Bayesian inference of both the dynamic latent states and the static parameters of the model. We build up on recent advances in compu…
An infinite parallel tempering bouncy particle sampler improves sampling efficiency for multimodal distributions.
problem Sampling from complex posterior distributions with high accuracy and efficiency.
method Introduced an infinite parallel tempering bouncy particle sampler (BPS-PT) to accelerate convergence.
result Demonstrated improved sampling efficiency for multimodal distributions through numerical simulations.
A new method for online VI in SSMs using asymptotic contrast.
problem Lack of functionality for streaming data in standard VI methods for SSMs.
method Propose maximising an IWAE-type variational lower bound on the asymptotic contrast function using stochastic approximation.
result OSIWAE allows for online learning of model parameters and latent states.
In Bayesian inference, the posterior distributions are difficult to obtain analytically for complex models such as neural networks. Variational inference usually uses a parametric distribution for approximation, from which we can easily draw samples. Recently discrete approximation by particles has attracted attention …
Two SVGD variants achieve fast convergence with provable guarantees.
problem Understanding and improving SVGD's performance with finite particles.
method Introducing virtual particles and novel stochastic approximations.
result Provable fast convergence rates for finite-particle SVGD variants.
Stein variational neural network ensembles improve diversity and uncertainty estimation.
problem Lack of proper Bayesian justification and diversity guarantees in deep neural network ensembles.
method Particle-based inference methods, specifically Stein variational gradient descent (SVGD), operating in weight space, function space, and hybrid settings.
result SVGD methods improve diversity and uncertainty estimation, approaching the true Bayesian posterior more closely.
SVI and GP surrogates improve calibration of ABMs in epidemiology.
problem Calibrating stochastic ABMs in epidemiology is computationally expensive.
method Stein Variational Inference (SVI) with Gaussian process (GP) surrogates.
result SVI maintains comparable predictive accuracy and calibration effectiveness to MCMC.
ASVGD accelerates SVGD for efficient sampling.
problem Slow SVGD in high-dimensional sampling.
method Accelerated gradient flow in a metric space of probability densities, using Nesterov's method and momentum-based updates.
result ASVGD outperforms SVGD and other methods in sampling efficiency.