Improved sampling method using regularized Stein Variational Gradient Flow.
problem Improving the accuracy of sampling methods in machine learning.
method Proposed Regularized Stein Variational Gradient Flow to interpolate between SVGD and Wasserstein Gradient Flow.
result Established theoretical properties and provided preliminary numerical evidence of improved performance.
Stein's method improves probabilistic inference and learning.
problem Improving probabilistic inference and learning methods.
method Constructing Stein discrepancies from Stein operators and Stein sets, discussing their properties.
result Connection between Stein operators and Stein variational gradient descent.
A new method reduces complexity and uncertainty in neural networks.
problem Uncertainty quantification in complex neural networks.
method Condensed Stein Variational Gradient Descent (cSVGD) method.
result Condensed SVGD provides uncertainty quantification on parameters.
Improved neural network ensembles using Stein Variational Newton updates.
problem Lack of efficient second-order information in current ensemble methods.
method Proposes a novel approximate Bayesian inference method integrating Stein Variational Newton updates with scalable Hessian approximations.
result Significantly faster convergence and more accurate posterior distribution approximations.
BSVGD improves sampling for multimodal distributions using branching.
problem Sampling from multimodal distributions.
method Random branching in Stein Variational Gradient Descent (SVGD).
result Theoretical convergence guarantee and empirical validation.
We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian inference method that generalizes Stein Variational Gradient Descent (SVGD) to Riemann manifold. The benefits are two-folds: (i) for inference tasks in Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information geometry, and …
SVGD algorithm converges at rate 1/sqrt(log log n) for sub-Gaussian distributions.
problem Approximating a probability distribution with particles.
method Stein variational gradient descent (SVGD) with finite particles and sub-Gaussian target distribution.
result SVGD achieves a convergence rate of 1/sqrt(log log n) for sub-Gaussian distributions.
New method improves sample diversity and efficiency from complex distributions.
problem Sampling from intractable un-normalized distributions with high auto-correlation.
method Stein self-repulsive dynamics using a repulsive force to push samples away from past trajectories.
result Significantly decreases auto-correlation and increases effective sample size.
Paper explores SVGD for Bayesian inference, linking deterministic and stochastic dynamics.
problem Bayesian inference and Markov chain Monte Carlo methods.
method Stein variational gradient descent (SVGD) with deterministic and stochastic dynamics.
result Identifies Stein-Fisher information as the leading order contribution in the long-time and many-particle regime.
Inspired by the seminal work on Stein Variational Inference and Stein Variational Policy Gradient, we derived a method to generate samples from the posterior variational parameter distribution by \textit{explicitly} minimizing the KL divergence to match the target distribution in an amortize fashion. Consequently, we a…
New samplers minimize KL divergence for constrained and non-Euclidean geometries.
problem Efficient sampling from constrained and non-Euclidean distributions.
method Stein Variational Mirror Descent and Mirrored Stein Variational Gradient Descent.
result New samplers converge more rapidly and accurately than prior methods.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
New Stein operator improves robustness in model inference.
problem Improving robustness in inference for unnormalized models.
method Density-power weighted Stein operator (γ-Stein operator). result Robust methods for goodness-of-fit testing and posterior approximation.
Stein variational gradient descent improves inference in Gaussian process models.
problem Inference in Gaussian process models with non-Gaussian likelihoods and large data volumes is computationally intensive and inaccurate with traditional methods.
method Stein variational gradient descent (SVGD) for non-parametric inference.
result SVGD monotonically decreases the Kullback-Leibler divergence from the sampling distribution to the true posterior.
Sliced kernelized Stein discrepancy improves goodness-of-fit tests and model learning in high dimensions.
problem The curse-of-dimensionality in kernelized Stein discrepancy (KSD).
method Sliced Stein discrepancy and its scalable variants using optimal one-dimensional projections.
result Significantly outperforms KSD and baselines in goodness-of-fit tests and improves model learning.
SMI uses mixture models to improve SVGD's performance in Bayesian inference.
problem Variance collapse in SVGD for Bayesian inference, especially with small models.
method Generalizes SVGD to Stein mixture models, optimizing an ELBO lower bound.
result SMI avoids variance collapse and accurately estimates uncertainty for small BNNs.
Bayesian online changepoint detection (BOCPD) (Adams & MacKay, 2007) offers a rigorous and viable way to identify changepoints in complex systems. In this work, we introduce a Stein variational online changepoint detection (SVOCD) method to provide a computationally tractable generalization of BOCPD beyond the exponent…
Proposes a method to stabilize Black Box Variational Inference using the James-Stein estimator.
problem Stability issues and fine-tuning required in basic Black Box Variational Inference.
method Reframe stochastic gradient ascent as multivariate estimation problem using James-Stein estimator.
result Provides a simpler method with consistent performance in terms of model fit and convergence time.
New method uses multiple kernels to improve SVGD performance.
problem Sub-optimal performance of single kernel in SVGD.
method Combines multiple kernels to approximate optimal kernel, using Kernelized Stein Discrepancy (KSD) and constructing Multiple Kernel SVGD (MK-SVGD).
result Consistently matches or outperforms competing methods in experiments.
We propose a general purpose variational inference algorithm that forms a natural counterpart of gradient descent for optimization. Our method iteratively transports a set of particles to match the target distribution, by applying a form of functional gradient descent that minimizes the KL divergence. Empirical studies…
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
Stein variational neural network ensembles improve diversity and uncertainty estimation.
problem Lack of proper Bayesian justification and diversity guarantees in deep neural network ensembles.
method Particle-based inference methods, specifically Stein variational gradient descent (SVGD), operating in weight space, function space, and hybrid settings.
result SVGD methods improve diversity and uncertainty estimation, approaching the true Bayesian posterior more closely.
The paper analyzes rates for a modified gradient descent method using Stein variational gradients.
problem Improving the accuracy of gradient descent methods for complex target distributions.
method Derives finite-particle rates for regularized Stein variational gradient descent (R-SVGD).
result Establishes explicit non-asymptotic bounds for time-averaged empirical measures.
Efficiently samples and learns densities with symmetries using equivariant methods.
problem Efficiently sampling and learning densities with symmetries.
method Equivariant Stein Variational Gradient Descent (SVGD) and equivariant energy based models.
result Improves and scales up training of energy based models.
A method for efficient approximate inference on discrete distributions.
problem Applying SVGD to discrete distributions.
method Transforming discrete distributions to piecewise continuous distributions for SVGD application.
result Outperforms traditional algorithms and ensemble methods on discrete graphical models.
Stein variational gradient descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate given distributions, based on an efficient gradient-based update that guarantees to optimally decrease the KL divergence within a function space. This paper develops the first th…
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effe…
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
New algorithm improves convergence of Bayesian inference.
problem Efficient sampling from complex distributions.
method Stochastic Stein Variational Newton method (sSVN).
result sSVN converges faster and more accurately than SVGD.
Paper analyzes SVGD algorithm for non-asymptotic convergence.
problem Optimizing a set of particles to approximate a target probability distribution.
method Finite time analysis of SVGD algorithm, providing descent lemma and convergence rates.
result SVGD algorithm decreases the objective at each iteration and converges to the target distribution.
A new method for SVGD reduces variance in high dimensions.
problem High-dimensional variance in SVGD.
method Grassmann Stein Variational Gradient Descent (GSVGD) projects onto arbitrary subspaces and uses coupled Grassmann-valued diffusion.
result GSVGD explores high-dimensional problems with intrinsic low-dimensional structure efficiently.
Ad-SVGD optimizes kernel parameters for SVGD, improving inference performance.
problem Efficiently approximating posterior distributions in Bayesian inference.
method Adaptive kernel selection for SVGD dynamics.
result Ad-SVGD outperforms standard heuristics in various tasks.
We propose in this work RBM-SVGD, a stochastic version of Stein Variational Gradient Descent (SVGD) method for efficiently sampling from a given probability measure and thus useful for Bayesian inference. The method is to apply the Random Batch Method (RBM) for interacting particle systems proposed by Jin et al to the …
ASVGD accelerates SVGD for efficient sampling.
problem Slow SVGD in high-dimensional sampling.
method Accelerated gradient flow in a metric space of probability densities, using Nesterov's method and momentum-based updates.
result ASVGD outperforms SVGD and other methods in sampling efficiency.
Bayesian deep learning improves maintenance planning uncertainty quantification.
problem Estimating the remaining useful life of physical systems with uncertainty quantification.
method Stein variational gradient descent for training Bayesian neural networks.
result Bayesian deep learning models trained via Stein variational gradient descent outperform other methods in convergence speed and predictive performance.
Improves SVGD for high-dimensional Bayesian inference by reducing variance collapse.
problem Variance collapse in SVGD reduces accuracy and diversity of estimation.
method Augmented Message Passing SVGD (AUMP-SVGD) method, a two-stage optimization procedure.
result AUMP-SVGD achieves satisfactory accuracy and overcomes variance collapse in various benchmark problems.
Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…
SBS uses SVGD to optimize continuous functions globally.
problem Global optimization of continuous Sobolev functions.
method Stein Boltzmann Sampling (SBS) with SVGD.
result SBS and its variants are highly competitive in global optimization.
New method reduces variance in Bayesian inverse problems.
problem High variance in Monte Carlo estimates for inverse problems.
method Conditional neural control variates based on Stein's identity.
result Substantial variance reduction across different inverse problems.
Improved convergence rates for Stein Variational Gradient Descent in finite-particle settings.
problem Improving convergence rates for Stein Variational Gradient Descent in finite-particle settings.
method Analyzing the time derivative of relative entropy and splitting it into dominant and smaller parts.
result Finite-particle convergence rates of order 1/\sqrt{N} for Kernelized Stein Discrepancy and Wasserstein-2 metrics.
Stein variational gradient descent (SVGD) is a non-parametric inference algorithm that evolves a set of particles to fit a given distribution of interest. We analyze the non-asymptotic properties of SVGD, showing that there exists a set of functions, which we call the Stein matching set, whose expectations are exactly …
We propose a simple algorithm to train stochastic neural networks to draw samples from given target distributions for probabilistic inference. Our method is based on iteratively adjusting the neural network parameters so that the output changes along a Stein variational gradient direction (Liu & Wang, 2016) that maxima…
A new method improves inference for complex Bayesian models.
problem Bayesian inference for doubly intractable distributions is computationally challenging.
method Monte Carlo Stein variational gradient descent (MC-SVGD) approach.
result The method achieves substantial computational gains over existing algorithms.
Stein transport improves Bayesian inference with faster convergence and reduced variance.
problem Efficiently approximating posterior distributions in Bayesian inference.
method A novel Bayesian inference method using Stein transport, which pushes particles along a curve of tempered distributions.
result Stein transport reaches posterior approximations faster and more accurately than Stein variational gradient descent (SVGD).
Stochastic Stein Discrepancies improve inference efficiency.
problem Intractable computation of Stein discrepancies.
method Subsampled approximations of Stein operators.
result Stochastic Stein Discrepancies inherit convergence properties of standard SDs.
We propose a novel adaptive importance sampling algorithm which incorporates Stein variational gradient decent algorithm (SVGD) with importance sampling (IS). Our algorithm leverages the nonparametric transforms in SVGD to iteratively decrease the KL divergence between our importance proposal and the target distributio…
A new method for diverse Pareto solutions in multi-objective learning.
problem Maximizing diversity while maximizing hypervolume in Pareto solutions.
method Annealed Stein Variational Gradient Descent (SVGD) with diverse gradient directions.
result SVH-MOL achieves superior performance in multi-objective and multi-task learning.
Uniform-in-time analysis for Stein Variational Gradient Descent across various metrics.
problem Understanding long-term behavior of finite-particle systems in relation to their mean-field limits.
method Developed uniform-in-time propagation-of-chaos results for continuous-time SVGD using cutoff strategies and finite-dimensional theories.
result Uniform-in-time propagation-of-chaos bounds in various metrics, including Langevin kernel Stein discrepancy, Wasserstein-1, and Wasserstein-2 distances.