Improved sampling method using regularized Stein Variational Gradient Flow.
problem Improving the accuracy of sampling methods in machine learning.
method Proposed Regularized Stein Variational Gradient Flow to interpolate between SVGD and Wasserstein Gradient Flow.
result Established theoretical properties and provided preliminary numerical evidence of improved performance.
A new method reduces complexity and uncertainty in neural networks.
problem Uncertainty quantification in complex neural networks.
method Condensed Stein Variational Gradient Descent (cSVGD) method.
result Condensed SVGD provides uncertainty quantification on parameters.
BSVGD improves sampling for multimodal distributions using branching.
problem Sampling from multimodal distributions.
method Random branching in Stein Variational Gradient Descent (SVGD).
result Theoretical convergence guarantee and empirical validation.
Stein's method improves probabilistic inference and learning.
problem Improving probabilistic inference and learning methods.
method Constructing Stein discrepancies from Stein operators and Stein sets, discussing their properties.
result Connection between Stein operators and Stein variational gradient descent.
Stein variational gradient descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate given distributions, based on an efficient gradient-based update that guarantees to optimally decrease the KL divergence within a function space. This paper develops the first th…
Paper analyzes SVGD algorithm for non-asymptotic convergence.
problem Optimizing a set of particles to approximate a target probability distribution.
method Finite time analysis of SVGD algorithm, providing descent lemma and convergence rates.
result SVGD algorithm decreases the objective at each iteration and converges to the target distribution.
Paper explores SVGD for Bayesian inference, linking deterministic and stochastic dynamics.
problem Bayesian inference and Markov chain Monte Carlo methods.
method Stein variational gradient descent (SVGD) with deterministic and stochastic dynamics.
result Identifies Stein-Fisher information as the leading order contribution in the long-time and many-particle regime.
SVGD algorithm converges at rate 1/sqrt(log log n) for sub-Gaussian distributions.
problem Approximating a probability distribution with particles.
method Stein variational gradient descent (SVGD) with finite particles and sub-Gaussian target distribution.
result SVGD achieves a convergence rate of 1/sqrt(log log n) for sub-Gaussian distributions.
Inspired by the seminal work on Stein Variational Inference and Stein Variational Policy Gradient, we derived a method to generate samples from the posterior variational parameter distribution by \textit{explicitly} minimizing the KL divergence to match the target distribution in an amortize fashion. Consequently, we a…
The paper analyzes rates for a modified gradient descent method using Stein variational gradients.
problem Improving the accuracy of gradient descent methods for complex target distributions.
method Derives finite-particle rates for regularized Stein variational gradient descent (R-SVGD).
result Establishes explicit non-asymptotic bounds for time-averaged empirical measures.
A new method for Bayesian inference in high dimensions using projected Stein variational gradient descent.
problem Bayesian inference challenges in high-dimensional data.
method Adapting Stein variational gradient descent to exploit intrinsic low dimensionality of data.
result pSVGD is more accurate and efficient than SVGD, especially in high-dimensional settings.
We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian inference method that generalizes Stein Variational Gradient Descent (SVGD) to Riemann manifold. The benefits are two-folds: (i) for inference tasks in Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information geometry, and …
Stein variational gradient descent improves inference in Gaussian process models.
problem Inference in Gaussian process models with non-Gaussian likelihoods and large data volumes is computationally intensive and inaccurate with traditional methods.
method Stein variational gradient descent (SVGD) for non-parametric inference.
result SVGD monotonically decreases the Kullback-Leibler divergence from the sampling distribution to the true posterior.
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
New method improves sample diversity and efficiency from complex distributions.
problem Sampling from intractable un-normalized distributions with high auto-correlation.
method Stein self-repulsive dynamics using a repulsive force to push samples away from past trajectories.
result Significantly decreases auto-correlation and increases effective sample size.
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
A new method for SVGD reduces variance in high dimensions.
problem High-dimensional variance in SVGD.
method Grassmann Stein Variational Gradient Descent (GSVGD) projects onto arbitrary subspaces and uses coupled Grassmann-valued diffusion.
result GSVGD explores high-dimensional problems with intrinsic low-dimensional structure efficiently.
New samplers minimize KL divergence for constrained and non-Euclidean geometries.
problem Efficient sampling from constrained and non-Euclidean distributions.
method Stein Variational Mirror Descent and Mirrored Stein Variational Gradient Descent.
result New samplers converge more rapidly and accurately than prior methods.
Improved convergence rates for Stein Variational Gradient Descent in finite-particle settings.
problem Improving convergence rates for Stein Variational Gradient Descent in finite-particle settings.
method Analyzing the time derivative of relative entropy and splitting it into dominant and smaller parts.
result Finite-particle convergence rates of order 1/\sqrt{N} for Kernelized Stein Discrepancy and Wasserstein-2 metrics.
We propose a general purpose variational inference algorithm that forms a natural counterpart of gradient descent for optimization. Our method iteratively transports a set of particles to match the target distribution, by applying a form of functional gradient descent that minimizes the KL divergence. Empirical studies…
SMI uses mixture models to improve SVGD's performance in Bayesian inference.
problem Variance collapse in SVGD for Bayesian inference, especially with small models.
method Generalizes SVGD to Stein mixture models, optimizing an ELBO lower bound.
result SMI avoids variance collapse and accurately estimates uncertainty for small BNNs.
Proposes a method to stabilize Black Box Variational Inference using the James-Stein estimator.
problem Stability issues and fine-tuning required in basic Black Box Variational Inference.
method Reframe stochastic gradient ascent as multivariate estimation problem using James-Stein estimator.
result Provides a simpler method with consistent performance in terms of model fit and convergence time.
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effe…
Improves gradient estimation for discrete distributions with variance reduction techniques.
problem Excessive variance in gradient estimation for discrete distributions.
method Stein operators for discrete distributions and control variates.
result Substantially lower variance in gradient estimation.
Efficiently samples and learns densities with symmetries using equivariant methods.
problem Efficiently sampling and learning densities with symmetries.
method Equivariant Stein Variational Gradient Descent (SVGD) and equivariant energy based models.
result Improves and scales up training of energy based models.
A method for efficient approximate inference on discrete distributions.
problem Applying SVGD to discrete distributions.
method Transforming discrete distributions to piecewise continuous distributions for SVGD application.
result Outperforms traditional algorithms and ensemble methods on discrete graphical models.
New Stein operator improves robustness in model inference.
problem Improving robustness in inference for unnormalized models.
method Density-power weighted Stein operator (γ-Stein operator). result Robust methods for goodness-of-fit testing and posterior approximation.
ASVGD accelerates SVGD for efficient sampling.
problem Slow SVGD in high-dimensional sampling.
method Accelerated gradient flow in a metric space of probability densities, using Nesterov's method and momentum-based updates.
result ASVGD outperforms SVGD and other methods in sampling efficiency.
Ad-SVGD optimizes kernel parameters for SVGD, improving inference performance.
problem Efficiently approximating posterior distributions in Bayesian inference.
method Adaptive kernel selection for SVGD dynamics.
result Ad-SVGD outperforms standard heuristics in various tasks.
Paper proposes LSVGD to stabilize GAN training via Langevin Stein Variational Gradient Descent.
problem Mode collapse and performance deterioration in GAN training.
method Langevin Stein Variational Gradient Descent (LSVGD) incorporating noise to stabilize training.
result LSVGD improves performance and stability of various GAN models.
Uniform-in-time analysis for Stein Variational Gradient Descent across various metrics.
problem Understanding long-term behavior of finite-particle systems in relation to their mean-field limits.
method Developed uniform-in-time propagation-of-chaos results for continuous-time SVGD using cutoff strategies and finite-dimensional theories.
result Uniform-in-time propagation-of-chaos bounds in various metrics, including Langevin kernel Stein discrepancy, Wasserstein-1, and Wasserstein-2 distances.
Stein variational gradient descent (SVGD) is a non-parametric inference algorithm that evolves a set of particles to fit a given distribution of interest. We analyze the non-asymptotic properties of SVGD, showing that there exists a set of functions, which we call the Stein matching set, whose expectations are exactly …
New method uses multiple kernels to improve SVGD performance.
problem Sub-optimal performance of single kernel in SVGD.
method Combines multiple kernels to approximate optimal kernel, using Kernelized Stein Discrepancy (KSD) and constructing Multiple Kernel SVGD (MK-SVGD).
result Consistently matches or outperforms competing methods in experiments.
Bayesian deep learning improves maintenance planning uncertainty quantification.
problem Estimating the remaining useful life of physical systems with uncertainty quantification.
method Stein variational gradient descent for training Bayesian neural networks.
result Bayesian deep learning models trained via Stein variational gradient descent outperform other methods in convergence speed and predictive performance.
DSVGD improves federated learning with fewer communication rounds.
problem Federated learning scalability and trustworthiness.
method Distributed Stein Variational Gradient Descent (DSVGD) for non-parametric Bayesian inference.
result DSVGD achieves comparable accuracy and scalability to other methods, with well-calibrated predictions.
Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…
Sliced kernelized Stein discrepancy improves goodness-of-fit tests and model learning in high dimensions.
problem The curse-of-dimensionality in kernelized Stein discrepancy (KSD).
method Sliced Stein discrepancy and its scalable variants using optimal one-dimensional projections.
result Significantly outperforms KSD and baselines in goodness-of-fit tests and improves model learning.
We propose in this work RBM-SVGD, a stochastic version of Stein Variational Gradient Descent (SVGD) method for efficiently sampling from a given probability measure and thus useful for Bayesian inference. The method is to apply the Random Batch Method (RBM) for interacting particle systems proposed by Jin et al to the …
Improves SVGD for high-dimensional Bayesian inference by reducing variance collapse.
problem Variance collapse in SVGD reduces accuracy and diversity of estimation.
method Augmented Message Passing SVGD (AUMP-SVGD) method, a two-stage optimization procedure.
result AUMP-SVGD achieves satisfactory accuracy and overcomes variance collapse in various benchmark problems.
Stein transport improves Bayesian inference with faster convergence and reduced variance.
problem Efficiently approximating posterior distributions in Bayesian inference.
method A novel Bayesian inference method using Stein transport, which pushes particles along a curve of tempered distributions.
result Stein transport reaches posterior approximations faster and more accurately than Stein variational gradient descent (SVGD).
This work improves texture segmentation by automatically tuning hyperparameters for Total-Variation.
problem The challenge is to automatically select hyperparameters for Total-Variation texture segmentation.
method The approach involves extending Stein's unbiased gradient estimator to handle correlated Gaussian noise, leading to an automatic tuning method.
result The method provides an automatic way to select hyperparameters for Total-Variation texture segmentation.
Stein variational neural network ensembles improve diversity and uncertainty estimation.
problem Lack of proper Bayesian justification and diversity guarantees in deep neural network ensembles.
method Particle-based inference methods, specifically Stein variational gradient descent (SVGD), operating in weight space, function space, and hybrid settings.
result SVGD methods improve diversity and uncertainty estimation, approaching the true Bayesian posterior more closely.
SRF improves kernel approximation and GP regression performance.
problem Efficient kernel approximation and Bayesian kernel learning in large-scale regression problems.
method Stein variational gradient descent to generate high-quality random features and approximate spectral measure posteriors.
result SRF outperforms traditional approaches in kernel approximation and GP regression.
Stein variational gradient decent (SVGD) has been shown to be a powerful approximate inference algorithm for complex distributions. However, the standard SVGD requires calculating the gradient of the target density and cannot be applied when the gradient is unavailable. In this work, we develop a gradient-free variant …
New algorithm improves convergence of Bayesian inference.
problem Efficient sampling from complex distributions.
method Stochastic Stein Variational Newton method (sSVN).
result sSVN converges faster and more accurately than SVGD.
Paper analyzes convergence rates of mean-field SVGD method.
problem Establishing quantitative rates of convergence for mean-field SVGD.
method Quantitative analysis of mean-field SVGD dynamics on torus.
result Explicit polynomial convergence rates in L2-norm for Riesz-type kernels.
We study an interacting particle system in Rd motivated by Stein variational gradient descent [Q. Liu and D. Wang, NIPS 2016], a deterministic algorithm for sampling from a given probability density with unknown normalization. We prove that in the large particle limit the empirical measure of the particle s…