New method reduces variance in complex probabilistic model optimization.
problem High variance in stochastic optimisation of complex models.
method Use recognition network to approximate optimal control variate for each mini-batch.
result Sub-optimal variance reduction is improved with new approach.
DSVNP uses global and local latent variables for improved neural process predictions.
problem Limited expressiveness of vanilla neural processes in capturing target-specific local variation.
method Introduces DSVNP combining global and local latent variables for prediction.
result Competitive prediction performance in multi-output regression and uncertainty estimation.
Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to over-fitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of GPs, but inference in these models has proved challenging. Existing approaches to infe…
Many machine learning applications are based on data collected from people, such as their tastes and behaviour as well as biological traits and genetic data. Regardless of how important the application might be, one has to make sure individuals' identities or the privacy of the data are not compromised in the analysis.…
We introduce local expectation gradients which is a general purpose stochastic variational inference algorithm for constructing stochastic gradients through sampling from the variational distribution. This algorithm divides the problem of estimating the stochastic gradients over multiple variational parameters into sma…
A scalable GPLVM model using stochastic variational inference.
problem Scalable inference for Gaussian process latent variable models.
method Doubly stochastic formulation of Bayesian GPLVM with minibatch training.
result High-fidelity reconstructions in the presence of missing data.
Bayesian Neural Networks built block-by-block with uncertainty estimates.
problem Building interpretable and uncertainty-aware neural networks.
method Bayesian Neural Networks (BNNs) constructed using blocks, with doubly stochastic variational inference for posterior approximation.
result Uncertainty estimates provided for Bayesian Neural Networks.
Enhances DGPs with adaptive RKHS Fourier features for better non-stationary pattern modeling.
problem Capturing complex non-stationary patterns in non-linear dynamical systems.
method Integrates ODE-based RKHS Fourier features into DGPs using convolution operations for adaptive amplitude and phase modulation. Uses a doubly stochastic variational inference framework.
result Improved predictive performance across various regression tasks.
DGPs improve air quality inference from sparse data.
problem Accurate air quality monitoring in unmonitored areas.
method Deep Gaussian Processes with Doubly Stochastic Variational Inference.
result DGPs outperform state-of-the-art models in AQ inference.
A recurring problem when building probabilistic latent variable models is regularization and model selection, for instance, the choice of the dimensionality of the latent space. In the context of belief networks with latent variables, this problem has been adressed with Automatic Relevance Determination (ARD) employing…
We propose an efficient method for estimating covariate effects in doubly-stochastic spatial models.
problem Computational demands and restrictive assumptions in existing doubly-stochastic spatial models.
method Penalized regression method for estimating covariate effects in doubly-stochastic point processes.
result Consistency and asymptotic normality of the covariate effect estimates achieved despite model misspecification.
This technical report proves components consistency for the Doubly Stochastic Dirichlet Process with exponential convergence of posterior probability. We also present the fundamental properties for DSDP as well as inference algorithms. Simulation toy experiment and real-world experiment results for single and multi-clu…
DSIVI improves variational autoencoders by optimizing a proper lower bound on ELBO.
problem Improving variational autoencoders with implicit priors.
method Introducing DSIVI, a method that optimizes a proper lower bound on ELBO for models with semi-implicit priors and posteriors.
result DSIVI improves the performance of VampPrior, a state-of-the-art prior for variational autoencoders.
Undirected graphical models are applied in genomics, protein structure prediction, and neuroscience to identify sparse interactions that underlie discrete data. Although Bayesian methods for inference would be favorable in these contexts, they are rarely used because they require doubly intractable Monte Carlo sampling…
A new method improves inference for complex Bayesian models.
problem Bayesian inference for doubly intractable distributions is computationally challenging.
method Monte Carlo Stein variational gradient descent (MC-SVGD) approach.
result The method achieves substantial computational gains over existing algorithms.
Bayesian approach to data association using Gaussian processes.
problem Separating data from different generating processes.
method Fully Bayesian approach with Gaussian process priors for structure encoding and doubly stochastic variational inference.
result Efficient learning scheme for deep Gaussian process priors.
Doubly SGD improves convergence for intractable objective optimization.
problem Optimizing objectives in sum of intractable expectations.
method Doubly SGD with doubly stochastic gradients and independent minibatching.
result Established convergence of doubly SGD under general conditions, including dependent component gradient estimators.
Robustly infers manifold density and geometry under high-dimensional noise.
problem Inaccurate kernel density estimation under high-dimensional noise.
method Doubly stochastic normalization of Gaussian kernel.
result Robust tools for density estimation, noise magnitude estimation, and distance approximation.
Many matching, tracking, sorting, and ranking problems require probabilistic reasoning about possible permutations, a set that grows factorially with dimension. Combinatorial optimization algorithms may enable efficient point estimation, but fully Bayesian inference poses a severe challenge in this high-dimensional, di…
Proposes SDRG to adjust missingness in machine learning models.
problem Systemic missingness in observational data leads to biased parameter estimation.
method Introduces SDRG using two models: weight-corrected gradients and per-covariate control variates.
result Empirically demonstrates convergence in training image classifiers with missing data.
A new method for training deep Gaussian processes using stochastic imputation.
problem Efficiently training deep Gaussian processes with varying regimes or sharp changes.
method Stochastic imputation to transform DGPs into linked GPs for efficient training.
result The method produces fast and analytically tractable predictions from DGP emulators.
Proposes DR-ACI for causal effect intervals with temporal dependence.
problem Causal effect intervals under temporal dependence.
method Doubly robust adaptive conformal inference (DR-ACI).
result Constructs prediction intervals for causal effects.
Simplified tutorial on doubly robust learning for causal inference.
problem Challenges in applying doubly robust methods due to complexity and software barriers.
method Combines propensity score and outcome modeling for robust causal inference.
result Makes doubly robust learning accessible through simplified methodology and practical examples.
New method improves variational inference for hierarchical models.
problem Limited expressivity of variational distributions in Bayesian models.
method Importance weighted hierarchical variational inference.
result Superior performance in experiments compared to existing methods.
State-space models (SSMs) are a highly expressive model class for learning patterns in time series data and for system identification. Deterministic versions of SSMs (e.g. LSTMs) proved extremely successful in modeling complex time series data. Fully probabilistic SSMs, however, are often found hard to train, even for …
Deep kernel processes unify various models using Gram matrices and kernel functions.
problem Unified representation of various deep learning models.
method Defining deep kernel processes with progressively transformed Gram matrices and sampling from inverse Wishart distributions.
result Deep Gaussian processes, BNNs, infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes.
Robust Bayesian changepoint detection with β-divergences reduces false discovery rates.
problem Detecting changepoints in non-stationary streaming data with high accuracy.
method Doubly robust Bayesian Online Changepoint Detection (BOCD) using β-divergences. result False discovery rates of changepoints reduced from over 90% to 0%.
We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Usin…
New method estimates causal effects in complex spaces using topological structures.
problem Challenges in estimating causal effects in non-Euclidean spaces.
method Developed a topological causal inference framework using power-weighted silhouette functions of persistence diagrams.
result Successfully quantifies topological treatment effects across various complex outcomes.
Stochastic variational inference for collapsed models has recently been successfully applied to large scale topic modelling. In this paper, we propose a stochastic collapsed variational inference algorithm in the sequential data setting. Our algorithm is applicable to both finite hidden Markov models and hierarchical D…
DGPs with variational inference suffer from SNR issues that degrade gradient estimates, leading to unreliable training.
problem SNR issues in gradient estimates for DGPs with variational inference.
method Adapted doubly reparameterized gradient estimators for DGP training.
result Fix improves predictive performance of DGP models.
Doubly-stochastic normalization improves robustness to heteroskedastic noise.
problem Robustness to heteroskedastic noise in affinity matrix construction.
method Doubly-stochastic normalization of the Gaussian kernel.
result Doubly-stochastic normalization converges to clean matrix with rate m−1/2 under heteroskedastic noise. Two new estimators improve VAE training for hierarchical and prior parameters.
problem Efficient gradient estimation for VAEs with hierarchical and prior parameters.
method Developed two generalizations of Doubly-Reparameterized Gradient Estimators (DReGs) for VAEs.
result Improved training of conditional and hierarchical VAEs on image modeling tasks.
Paper uses correlated samples to improve variance in variational inference.
problem High variance in stochastic variational inference.
method Differentiable antithetic sampling to generate correlated samples.
result Effective variance reduction in deep generative model learning.
Variational inference improves neural network matrix factorization for stochastic blockmodels.
problem Improving predictive performance of neural network matrix factorization for stochastic blockmodels.
method Construct Bayesian neural networks and fit with variational inference.
result Variational inference can achieve equivalent performance to neural networks on Movielens data.
Paper proposes a second-order method for faster SVI convergence.
problem Poor convergence rate of first-order SVI algorithms.
method Derives Hessian matrix and implements two numerical schemes for efficient second-order SVI.
result Proposed approach achieves faster convergence compared to first-order SVI.
Improves VAE training by refining variational parameters with BSVI.
problem Amortized inference in VAEs leads to suboptimal variational parameters and the amortization gap.
method Proposes BSVI, a refinement procedure using SVI's importance weights.
result Training VAEs with BSVI yields improved performance compared to SVI.
Unified framework connects stochastic optimization to Bayesian inference.
problem Stochastic optimization algorithms and their theoretical underpinnings.
method Latent variational problem and Forward Backward Stochastic Differential Equations (FBSDE).
result Recovery of various adaptive stochastic gradient descent methods.
New guarantees for black-box variational inference methods.
problem Insufficient theoretical guarantees for black-box variational inference.
method Novel convergence guarantees for stochastic optimization of variational inference.
result Provable convergence of proximal and projected stochastic gradient descent for variational inference.
We model messaging activities as a hierarchical doubly stochastic point process with three main levels, and develop an iterative algorithm for inferring actors' relative latent positions from a stream of messaging activity data. Each of the message-exchanging actors is modeled as a process in a latent space. The actors…
This paper uses Bayesian ARD to automatically determine utility functions for discrete choice models.
problem Challenging and time-consuming task in identifying optimal utility function specifications.
method Bayesian framework and automatic relevance determination (ARD) for data-driven utility function specification.
result The proposed DCM-ARD model accurately recovers true utility function specifications and outperforms previous methods.
Variational inference has experienced a recent surge in popularity owing to stochastic approaches, which have yielded practical tools for a wide range of model classes. A key benefit is that stochastic variational inference obviates the tedious process of deriving analytical expressions for closed-form variable updates…
DLFM models complex systems with uncertainty, outperforming traditional methods.
problem Modeling highly nonlinear dynamical systems with robust uncertainty quantification.
method Deep latent force model (DLFM) using physics-informed kernels derived from ODEs.
result DLFM achieves comparable performance to non-physics-informed models on univariate tasks and captures dynamics in real-world data.
Improves understanding of stochastic NGVI convergence rates.
problem Lack of knowledge about non-asymptotic convergence rates in stochastic NGVI.
method Proved non-asymptotic convergence rates for conjugate likelihoods and showed implicit optimization for non-conjugate likelihoods.
result First O(T1) non-asymptotic convergence rate for stochastic NGVI in conjugate likelihoods. Variational inference approximates the posterior distribution of a probabilistic model with a parameterized density by maximizing a lower bound for the model evidence. Modern solutions fit a flexible approximation with stochastic gradient descent, using Monte Carlo approximation for the gradients. This enables variatio…
MetaVAE learns transferable latent representations across related distributions.
problem Generative models struggle to adapt to new distributions.
method Doubly-amortized variational inference sharing computation across related models.
result MetaVAE significantly outperforms baselines on image classification tasks.
We introduce incremental variational inference and apply it to latent Dirichlet allocation (LDA). Incremental variational inference is inspired by incremental EM and provides an alternative to stochastic variational inference. Incremental LDA can process massive document collections, does not require to set a learning …
We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick. At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stat…