We introduce overdispersed black-box variational inference, a method to reduce the variance of the Monte Carlo estimator of the gradient in black-box variational inference. Instead of taking samples from the variational distribution, we use importance sampling to take samples from an overdispersed distribution in the s…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “black-box variational methods”
New method reduces inference variance for faster optimization.
problem High variance in black-box variational inference.
method Joint control variate addressing both data subsampling and Monte Carlo noise.
result Significantly reduced gradient variance, leading to faster optimization.
New guarantees for black-box variational inference methods.
problem Insufficient theoretical guarantees for black-box variational inference.
method Novel convergence guarantees for stochastic optimization of variational inference.
result Provable convergence of proximal and projected stochastic gradient descent for variational inference.
Variational inference has become a widely used method to approximate posteriors in complex latent variables models. However, deriving a variational inference algorithm generally requires significant model-specific analysis, and these efforts can hinder and deter us from quickly developing and exploring a variety of mod…
Paper advances black-box VI using flows and Monte-Carlo methods.
problem Improving automatic posterior inference in black-box VI.
method Combines normalizing flows, Monte-Carlo methods, and optimization considerations.
result Significant improvement in state-of-the-art variational inference.
Proposes a method to stabilize Black Box Variational Inference using the James-Stein estimator.
problem Stability issues and fine-tuning required in basic Black Box Variational Inference.
method Reframe stochastic gradient ascent as multivariate estimation problem using James-Stein estimator.
result Provides a simpler method with consistent performance in terms of model fit and convergence time.
Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriorscs.LG
DDVI uses diffusion models for variational inference, improving latent variable model performance.
problem Improving variational inference in latent variable models.
method Introduces diffusion-based variational posteriors trained with a regularized ELBO.
result Outperforms alternative variational posteriors on various benchmarks and a biology task.
New framework for variational coresets simplifies Bayesian inference for complex models.
problem Efficient Bayesian inference for complex models like neural networks.
method Black-box variational inference for coresets that handle intractable posterior distributions.
result Principled application of variational coresets to Bayesian neural networks.
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
Improves reliability of BBVI optimization methods.
problem Reliability issues and expertise required for BBVI optimization.
method RABVI framework with automated learning rate adjustment and KL divergence estimation.
result RABVI detects inaccurate variational approximations and optimizes reliability.
The paper introduces structured variational families to improve scalability in black-box variational inference.
problem Scalability issues in black-box variational inference, especially for large datasets and hierarchical models.
method Developed structured variational families that achieve better iteration complexity of O(N) compared to full-rank families.
result Structured variational families can achieve better scaling with respect to dataset size N, improving iteration complexity from O(N^2) to O(N).
VERA uses variational inference to jailbreak LLMs without manual optimization.
problem Lack of principled objective for gradient-based optimization in jailbreaking LLMs.
method VERA casts black-box jailbreak prompting as a variational inference problem, training a small attacker LLM to approximate the target LLM's posterior over adversarial prompts.
result VERA achieves strong performance across various target LLMs, demonstrating the value of probabilistic inference for adversarial prompt generation.
Guarantees convergence for black-box variational inference without modifications.
problem Convergence guarantees for black-box variational inference.
method Analysis of log-smooth posterior densities, location-scale variational family, and convergence rates of algorithm design choices.
result Proximal stochastic gradient descent fixes suboptimal convergence rates and achieves strongest known guarantees.
This work proposes a new method for variational inference using Wasserstein gradient descent.
problem Optimizing variational parameters to match a true posterior distribution.
method Reinterpreting VI as an optimization problem over a variational parameter space, using Wasserstein gradient descent.
result The proposed Wasserstein gradient descent can be seen as a generalization of existing optimization techniques in VI.
Improved inference for models with continuous latent variables.
problem Inference accuracy with traditional variational methods is limited.
method Reparameterized Variational Rejection Sampling (RVRS) using a proposal distribution with a reparameterized gradient estimator.
result RVRS offers a better trade-off between computational cost and inference fidelity.
QBVI uses natural gradients for efficient Bayesian learning.
problem Efficient Bayesian learning in complex models.
method Natural gradient updates in a black-box framework for exponential-family distributions.
result QBVI framework is effective for a wide range of Bayesian inference problems.
FlowVAT improves variational inference for multi-modal distributions.
problem Mode-seeking behavior and collapse in variational inference for complex posteriors.
method Conditional tempering approach for normalizing flow variational inference.
result FlowVAT outperforms traditional and adaptive annealing methods in multi-modal distributions, finding more modes and achieving better ELBO values.
Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiven…
Continuous latent time series models are prevalent in Bayesian modeling; examples include the Kalman filter, dynamic collaborative filtering, or dynamic topic models. These models often benefit from structured, non mean field variational approximations that capture correlations between time steps. Black box variational…
BBVI with STL converges geometrically under perfect specification, with quadratic variance bound.
problem Convergence rate of BBVI with STL estimator.
method Proved geometric convergence rate with quadratic variance bound for BBVI with STL estimator.
result BBVI with STL converges geometrically under perfect variational family specification.
Parameter inference for stochastic differential equations is challenging due to the presence of a latent diffusion process. Working with an Euler-Maruyama discretisation for the diffusion, we use variational inference to jointly learn the parameters and the diffusion paths. We use a standard mean-field variational appr…
Improves scalability and efficiency of mixture models in black-box variational inference.
problem Scaling mixture models in black-box variational inference leads to high parameter and time costs.
method Introduces MISVAE for amortized mixture parameter space and new ELBO estimators.
result Achieves superior estimation performance with fewer parameters and shorter inference time.
Recent variational inference methods use stochastic gradient estimators whose variance is not well understood. Theoretical guarantees for these estimators are important to understand when these methods will or will not work. This paper gives bounds for the common "reparameterization" estimators when the target is smoot…
We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trad…
In this work, we explore how probabilistic programs can be used to represent policies in sequential decision problems. In this formulation, a probabilistic program is a black-box stochastic simulator for both the problem domain and the agent. We relate classic policy gradient techniques to recently introduced black-box…
CO-BED optimizes experiments using Bayesian methods and information theory.
problem Optimizing experiments in a context-dependent manner.
method Formalizes contextual optimization with Bayesian experimental design, employing information-theoretic principles and black-box variational methods.
result CO-BED provides a general solution for contextual optimization problems.
Learning models with discrete latent variables using stochastic gradient descent remains a challenge due to the high variance of gradient estimates. Modern variance reduction techniques mostly consider categorical distributions and have limited applicability when the number of possible outcomes becomes large. In this w…
Paper introduces CageBO for optimizing complex public policy problems.
problem Complex decision-making and implicit constraints in public policy.
method CageBO framework using conditional variational autoencoder.
result CageBO outperforms baselines in optimizing large-scale police redistricting.
Black box variational inference allows researchers to easily prototype and evaluate an array of models. Recent advances allow such algorithms to scale to high dimensions. However, a central question remains: How to specify an expressive variational distribution that maintains efficient computation? To address this, we …
We propose a black-box algorithm called {\it Adversarial Variational Inference and Learning} (AdVIL) to perform inference and learning on a general Markov random field (MRF). AdVIL employs two variational distributions to approximately infer the latent variables and estimate the partition function of an MRF, respective…
Many computationally-efficient methods for Bayesian deep learning rely on continuous optimization algorithms, but the implementation of these methods requires significant changes to existing code-bases. In this paper, we propose Vprop, a method for Gaussian variational inference that can be implemented with two minor c…
MCMC-driven learningstat.ML
Unified framework for MCMC and machine learning problems.
problem Intersection of MCMC and machine learning problems.
method Unified framework integrating various MCMC and machine learning techniques.
result Translation and generalization of theory and methods.
Many probabilistic models of interest in scientific computing and machine learning have expensive, black-box likelihoods that prevent the application of standard techniques for Bayesian inference, such as MCMC, which would require access to the gradient or a large number of likelihood evaluations. We introduce here a n…
We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick. At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stat…
DADVI improves ADVI by using deterministic approximation for faster, more accurate posterior estimation.
problem Intractable posterior uncertainty estimates and lack of clear convergence criteria in ADVI.
method Replaces stochastic MFVB objective with deterministic Monte Carlo approximation (SAA) and uses second-order optimization.
result DADVI provides faster and more accurate posterior estimates with default settings.
A new variational inference method using Gaussian score matching.
problem Approximating posterior distributions in Bayesian statistics.
method Score matching principle applied to variational inference.
result Gaussian score matching VI (GSM-VI) is faster and requires fewer gradient evaluations.
BaM improves BBVI by optimizing a score-based divergence, leading to faster convergence.
problem Slow convergence of black-box variational inference methods.
method Batch and match (BaM) approach based on a score-based divergence.
result BaM converges exponentially quickly to the target mean and covariance.
New bounds show BBVI's gradient variance matches SGD conditions, improving parameterization efficiency.
problem Understanding and improving the convergence of black-box variational inference (BBVI).
method Showed BBVI satisfies matching gradient variance bounds corresponding to the ABC condition for smooth and quadratically-growing log-likelihoods.
result Proven BBVI's gradient variance matches SGD conditions, with superior dimensional dependence for mean-field parameterization.
VBMC+VIQR outperforms noisy models in Bayesian inference.
problem Bayesian inference with noisy likelihoods in complex models.
method Gaussian process surrogates, expected information gain, variational interquantile range.
result VBMC+VIQR achieves state-of-the-art performance in noisy inference benchmarks.
Post-process Bayesian inference speeds up posterior approximation.
problem Leveraging pre-existing model evaluations for quick posterior approximation.
method Variational Sparse Bayesian Quadrature (VSBQ) using sparse Gaussian process (GP) surrogate model.
result VSBQ builds high-quality posterior approximations from existing optimization traces.
BBVI converges nearly dimensionally independent for log-concave targets.
problem Efficiently optimizing variational parameters in high-dimensional spaces.
method Proved convergence rate of BBVI with reparametrization gradient for log-concave targets.
result BBVI converges with nearly independent dimension dependence for log-concave targets.
We present the Variational Adaptive Newton (VAN) method which is a black-box optimization method especially suitable for explorative-learning tasks such as active learning and reinforcement learning. Similar to Bayesian methods, VAN estimates a distribution that can be used for exploration, but requires computations th…
AdvFlow generates imperceptible adversarial images using normalizing flows.
problem Adversarial attacks on deep learning classifiers.
method Normalizing flows to model adversarial examples.
result Generated adversaries closely follow clean data distribution, making them harder to detect.
We investigate the optimization of two probabilistic generative models with binary latent variables using a novel variational EM approach. The approach distinguishes itself from previous variational approaches by using latent states as variational parameters. Here we use efficient and general purpose sampling procedure…
VES-Gamma adapts EI using information-theoretic principles.
problem Optimizing black-box functions using Bayesian optimization.
method Variational Entropy Search (VES) and VES-Gamma algorithm.
result VES-Gamma improves EI by incorporating information-theoretic concepts.
While stochastic variational inference is relatively well known for scaling inference in Bayesian probabilistic models, related methods also offer ways to circumnavigate the approximation of analytically intractable expectations. The key challenge in either setting is controlling the variance of gradient estimates: rec…
Black-box variational inference tries to approximate a complex target distribution though a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees require structural properties of the objective. This paper shows that for location-scale family approximations, if the targ…
Derives VMP for LDA, simplifying inference for topic modeling.
problem Manual derivation of VMP equations for LDA is challenging and time-consuming.
method Detailed derivation of VMP update equations for LDA.
result Enables easier implementation of VMP for LDA models.