Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
We consider the problem of sequential learning from categorical observations bounded in [0,1]. We establish an ordering between the Dirichlet posterior over categorical outcomes and a Gaussian posterior under observations with N(0,1) noise. We establish that, conditioned upon identical data with at least two observatio…
A new algorithm optimizes Gaussian process posterior mean functions efficiently.
problem Optimizing Gaussian process posterior mean functions over hyperrectangles is challenging due to nonlinearity and nonconvexity.
method PALM-Mean, a piecewise-analytic lower-bounding framework embedded in reduced-space spatial branch-and-bound.
result PALM-Mean improves scalability for large datasets compared to general-purpose solvers.
A new ensemble filter uses transport maps and MMD optimization for high-dimensional data assimilation.
problem High-dimensional data assimilation challenges in ensemble filtering.
method Optimized Maximum Mean Discrepancy (MMD) for transport map construction.
result Significant improvement in robustness and posterior approximation.
Study shows TAP free energy minimization provides better posterior inference in high-dimensional linear models.
problem Deviation from true posterior mean and underestimation of posterior uncertainty in variational inference.
method Minimization of TAP free energy in a high-dimensional asymptotic framework, showing geometric and statistical properties.
result Local minimizer of TAP free energy provides consistent estimate of posterior marginals and correctly calibrated posterior inference.
Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
Sparse variational approximations allow for principled and scalable inference in Gaussian Process (GP) models. In settings where several GPs are part of the generative model, theses GPs are a posteriori coupled. For many applications such as regression where predictive accuracy is the quantity of interest, this couplin…
Sparse matrices simplify computation of GP variances and likelihoods.
problem Efficient computation of posterior variance and log-likelihood for additive Matérn GPs.
method Represented posterior mean, variance, log-likelihood, and gradient using sparse matrices.
result Efficient computation of posterior mean, variance, log-likelihood, and gradient in O(nlogn) time. Geometric framework analyzes bias in variational inference for posterior functionals.
problem Analyzing the bias of posterior functionals under variational approximations.
method Developed a geometric framework to evaluate the bias of posterior functionals using the variational tangent space.
result The leading-order bias of a posterior functional is determined by its component orthogonal to the variational tangent space.
Bayesian inference for inverse problems using mean-shift interacting particles
problem Bayesian inference for inverse problems
method Amortized mean-shift interacting particles
result Improves accuracy of Bayesian inference by reducing the number of samples needed
MFVI can overestimate predictive variance compared to the exact posterior
problem MFVI underestimates posterior variance
method Analyzing conjugate Bayesian Linear Regression
result MFVI can overestimate predictive variance compared to the exact posterior
Robust VB framework handles contamination using min-max median aggregation.
problem Handling contamination and outliers in datasets.
method Partition data into subsets, formulate robust optimization problem, use min-max median KL divergence.
result Min-max median formulation improves robustness and statistical rates.
Compact parameterization improves Bayesian neural network performance.
problem Improving performance of Bayesian neural networks using variational methods.
method Restricting variational distribution to a k-tied Normal distribution with low-rank factorization.
result Compact parameterization improves signal-to-noise ratio and convergence speed.
Paper addresses the disparity between sampled and mean representations in disentangled learning.
problem Disparity between sampled and mean representations in disentangled learning.
method Proposes a method to eliminate the disparity by proving and utilizing the relationship between total correlation of sampled and mean representations for multivariate normal distributions.
result Demonstrates that a factorized mean representation can have lower total correlation than the sampled representation.
Advocates for a new posterior that predicts better than classical and generalised Bayes.
problem Combining parameter inference and density estimation for better predictive models.
method Predictively Oriented (PrO) posterior using mean field Langevin dynamics.
result PrO posteriors converge to the predictively optimal model average, adapting to model misspecification.
Proposes a new method for posterior sampling using MMD with negative distance kernel.
problem Posterior sampling and conditional generative modeling.
method Approximates joint distribution using discrete Wasserstein gradient flows of MMD with negative distance kernel.
result Establishes an error bound for posterior distributions and proves the method is a Wasserstein gradient flow.
We conduct non-asymptotic analysis on the mean-field variational inference for approximating posterior distributions in complex Bayesian models that may involve latent variables. We show that the mean-field approximation to the posterior can be well-approximated relative to the Kullback-Leibler divergence discrepancy m…
A new particle algorithm improves mean-field variational inference.
problem Efficiently approximating nonparametric posterior distributions in machine learning.
method Introduces PArticle VI (PAVI), a novel particle-based algorithm for nonparametric mean-field approximation.
result Obtains non-asymptotic error bounds for PArticle VI, providing the first end-to-end guarantee for particle-based MFVI.
Wide BNNs with odd activations fail to approximate data under mean-field inference.
problem Theoretical limitations of mean-field variational inference in wide, deep Bayesian neural networks.
method Analysis of mean-field variational inference in fully-connected BNNs with odd activation functions and Gaussian likelihood.
result The optimal mean-field variational posterior predictive distribution converges to the prior predictive distribution as network width increases.
FastMuyGPs speeds up GP predictions for large datasets.
problem High cost of Gaussian process predictions for large data.
method Combines cross-validation, batching, nearest neighbors sparsification, and precomputation.
result Superior accuracy and competitive runtime compared to other methods.
In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates. To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties. In this paper, we build a ps…
In the context of recent deep clustering studies, discriminative models dominate the literature and report the most competitive performances. These models learn a deep discriminative neural network classifier in which the labels are latent. Typically, they use multinomial logistic regression posteriors and parameter re…
Bayesian deep learning faces posterior collapse due to likelihood vs. prior competition.
problem Posterior collapse in Bayesian deep learning models.
method Identified competition between likelihood and prior regularization in a linear latent variable model.
result Posterior collapse is related to neural and dimensional collapse, suggesting a broader learning issue.
Differential privacy of Gaussian process posterior sampling
problem Privacy of posterior sample paths from Gaussian process
method Intrinsic randomness yields DP guarantees
result Intrinsic randomness yields DP guarantees
Fuses posterior distributions from different datasets using KL divergence.
problem Combining information from multiple datasets with uncertainty.
method Mean field assumption, KL divergence, assign-and-average approach.
result Efficient non-parametric algorithm for fused model computation.
New method quantifies uncertainty in denoising models.
problem Uncertainty quantification in denoising models.
method Derives a relation between posterior moments and derivatives, uses it for efficient uncertainty quantification.
result Efficient computation of principal components and full marginal distributions of the posterior.
We propose a general algorithm for approximating nonstandard Bayesian posterior distributions. The algorithm minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribution. Our method can be used to approximate any posterior distribution, provided that it is given i…
New algorithm improves Bayesian inference for complex models.
problem Poor performance of existing Bayesian approaches for simulators.
method Posterior bootstrap and maximum mean discrepancy estimators.
result Strong robustness and parallelizability of the new algorithm.
Kernel methods are one of the mainstays of machine learning, but the problem of kernel learning remains challenging, with only a few heuristics and very little theory. This is of particular importance in methods based on estimation of kernel mean embeddings of probability measures. For characteristic kernels, which inc…
DADVI improves ADVI by using deterministic approximation for faster, more accurate posterior estimation.
problem Intractable posterior uncertainty estimates and lack of clear convergence criteria in ADVI.
method Replaces stochastic MFVB objective with deterministic Monte Carlo approximation (SAA) and uses second-order optimization.
result DADVI provides faster and more accurate posterior estimates with default settings.
One of the core problems in variational inference is a choice of approximate posterior distribution. It is crucial to trade-off between efficient inference with simple families as mean-field models and accuracy of inference. We propose a variant of a greedy approximation of the posterior distribution with tractable bas…
Unified framework combines views and optimization for better portfolio management.
problem Optimizing portfolio weights with dynamic adjustment based on volatility.
method Dynamic sliding window adjusting horizon, factor estimates, BL posterior returns, and weights over time.
result Outperforms dynamic mean-variance optimization without BL views, providing stronger downside risk control.
Paper tackles model vulnerabilities by reconstructing training data.
problem Reconstructing training data from model parameters poses a security risk.
method Developed a mathematical framework and score matching method for both Bayesian and non-Bayesian models.
result First score matching framework for reconstructing data in Bayesian models.
A new data-adaptive prior stabilizes kernel learning in operators.
problem Learning kernels in operators from data is ill-posed due to nonlocal dependence.
method Introduces a data-adaptive prior to stabilize the Bayesian posterior mean.
result The data-adaptive prior achieves a stable posterior with small noise limits.
We consider the problem of subspace estimation in a Bayesian setting. Since we are operating in the Grassmann manifold, the usual approach which consists of minimizing the mean square error (MSE) between the true subspace U and its estimate U^ may not be adequate as the MSE is not the natural metric in the Gra…
Learning in Gaussian Process models occurs through the adaptation of hyperparameters of the mean and the covariance function. The classical approach entails maximizing the marginal likelihood yielding fixed point estimates (an approach called \textit{Type II maximum likelihood} or ML-II). An alternative learning proced…
TM-VI uses flexible transformation models to approximate complex posteriors in Bayesian models.
problem Approximating complex posteriors in Bayesian models with limited flexibility.
method Transformation models for variational inference (TM-VI).
result TM-VI allows accurate approximation of complex posteriors in models with one parameter and works in a mean-field fashion for multi-parameter models.
Paper proposes a method to break symmetries in Bayesian matrix factorization.
problem Symmetries in posterior distribution reduce MCMC sampling efficiency.
method Modification to Gaussian prior mean and covariance to break symmetries.
result Breaking symmetries leads to lower autocorrelation and reconstruction errors.
We study the relationship between online Gaussian process (GP) regression and kernel least mean squares (KLMS) algorithms. While the latter have no capacity of storing the entire posterior distribution during online learning, we discover that their operation corresponds to the assumption of a fixed posterior covariance…
Dynamic trees are mixtures of tree structured belief networks. They solve some of the problems of fixed tree networks at the cost of making exact inference intractable. For this reason approximate methods such as sampling or mean field approaches have been used. However, mean field approximations assume a factorized di…
Variational Bayes (VB) is a recent approximate method for Bayesian inference. It has the merit of being a fast and scalable alternative to Markov Chain Monte Carlo (MCMC) but its approximation error is often unknown. In this paper, we derive the approximation error of VB in terms of mean, mode, variance, predictive den…
Variational inference (VI) provides fast approximations of a Bayesian posterior in part because it formulates posterior approximation as an optimization problem: to find the closest distribution to the exact posterior over some family of distributions. For practical reasons, the family of distributions in VI is usually…
CNR uses convex optimization to estimate conditional distributions.
problem Estimating uncertainty in predictions and posterior conditional distributions.
method Convex optimization of a posterior defined via non-linear transformations on Gaussians.
result CNR can fit arbitrary conditional distributions, including multimodal and non-symmetric ones.
We identify a new variational inference scheme for dynamical systems whose transition function is modelled by a Gaussian process. Inference in this setting has either employed computationally intensive MCMC methods, or relied on factorisations of the variational posterior. As we demonstrate in our experiments, the fact…
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable cova…
This paper proposes a new algorithm for Gaussian process classification based on posterior linearisation (PL). In PL, a Gaussian approximation to the posterior density is obtained iteratively using the best possible linearisation of the conditional mean of the labels and accounting for the linearisation error. PL has s…
A new method improves Bayesian filtering in nonlinear systems.
problem Bayesian filtering in nonlinear dynamical systems with non-Gaussian posteriors.
method Transport maps with block-triangular structure and gradient flows for MMD minimization.
result Accurate approximation of non-Gaussian posteriors without particle collapse.
Bayesian SAE model with spectral clustering and uncertainty quantification.
problem Small Area Estimation (SAE) with uncertainty quantification.
method Spectral clustering with external covariates, posterior projections, and CPMSE.
result Closed form expressions for posterior mean estimators and CPMSE.