Review of priors in Bayesian deep learning models.
problem The importance of prior choices in Bayesian deep learning models.
method Overview of different priors and methods of learning priors from data.
result Motivate practitioners to think carefully about prior specification.
Proposes MOPED method for choosing priors in Bayesian DNNs.
problem Challenges in specifying meaningful priors for deep neural networks.
method Two-stage hierarchical modeling with empirical Bayes.
result MOPED enables scalable variational inference and reliable uncertainty quantification.
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
Stochastic variational inference is an established way to carry out approximate Bayesian inference for deep models. While there have been effective proposals for good initializations for loss minimization in deep learning, far less attention has been devoted to the issue of initialization of stochastic variational infe…
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
This paper improves deep learning by integrating Bayesian inference into network structure learning.
problem Bayesian inference in high-dimensional, over-parameterized neural networks.
method Developed an efficient stochastic variational inference approach to learn both network structure and weights.
result Empirically, the method exhibits competitive predictive performance and preserves Bayesian benefits.
A new method reduces over-regularization in Bayesian deep learning.
problem Bayesian inference for over-parameterized models is challenging.
method Walsh-Hadamard Variational Inference (WHVI) using factorization strategies.
result WHVI avoids over-regularization issues and yields speedups and model reductions.
Unified theory linking Bayesian and ensemble methods in deep learning.
problem Uncertainty quantification in deep learning.
method Reformulating optimisation as convex optimisation in probability measures, studying Wasserstein gradient flows.
result Unified theory explaining success of deep ensembles over variational inference.
Bayes-CATSI uses variational Bayesian deep learning for medical time series data imputation.
problem Missing values in medical time series data.
method Bayes-CATSI integrates variational inference for uncertainty quantification and context-aware imputation.
result Bayes-CATSI outperforms CATSI by 9.57% in imputation performance.
Bayesian deep learning improves maintenance planning uncertainty quantification.
problem Estimating the remaining useful life of physical systems with uncertainty quantification.
method Stein variational gradient descent for training Bayesian neural networks.
result Bayesian deep learning models trained via Stein variational gradient descent outperform other methods in convergence speed and predictive performance.
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
In this paper we introduce ZhuSuan, a python probabilistic programming library for Bayesian deep learning, which conjoins the complimentary advantages of Bayesian methods and deep learning. ZhuSuan is built upon Tensorflow. Unlike existing deep learning libraries, which are mainly designed for deterministic neural netw…
Improves Bayesian inference for deep models to better approximate posterior distributions.
problem Bayesian deep learning struggles with intractable posterior distributions, leading to overconfident predictions.
method Uses variational inference to approximate posterior distributions, proposing a unified view and improving inference for deep Gaussian processes.
result Variational inference can provide a lower bound for marginal likelihood, facilitating model selection and optimization.
Bayesian deep learning made practical with variational inference.
problem Impracticality of Bayesian methods in deep learning.
method Natural-gradient variational inference with techniques like batch normalisation, data augmentation, and distributed training.
result Achieves similar performance to Adam optimiser with fewer epochs, preserving Bayesian benefits.
Bayesian neural networks improve deep learning's accuracy and uncertainty estimation.
problem Overconfident predictions, adversarial attacks, and variability underestimation in deep models.
method Stochastic relaxation of feed-forward rectified neural networks with sparsity-promoting priors and Polya-Gamma data augmentation.
result Improved scalability and robustness to architectural design through approximate variational inference.
Proposes CVRCF for streaming recommender systems combining deep learning and probabilistic models.
problem Streaming recommendation problem with dynamic data and complexity.
method Coupled Variational Recurrent Collaborative Filtering (CVRCF) framework integrating stochastic processes and deep factorization models.
result Favorable performance in temporal dependency modeling and predictive accuracy compared to state-of-the-art methods.
Bayesian sparsification reduces deep neural network complexity.
problem Complexity of deep neural networks limits their performance.
method Combines Bayesian shrinkage priors with stochastic variational inference.
result Bayesian model reduction (BMR) is a more efficient alternative for pruning model weights.
Bayesian methods improve OoD detection in deep networks.
problem Detecting Out-of-Distribution (OoD) inputs in deep neural networks.
method Three Bayesian inference approaches applied to VAE weights.
result Improved OoD detection scores over benchmarks.
Bayesian methods enhance deep learning models by improving reliability and uncertainty.
problem Improving reliability and uncertainty awareness in deep learning models.
method Approximate Bayesian inference techniques, including SG-MCMC and VI, applied to deep learning models.
result Enhanced posterior inference for deep learning models, particularly in neural networks and generative models.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
Variational Bayes simplifies Bayesian neural networks for uncertainty quantification.
problem Quantifying uncertainty in neural networks' outputs.
method Approximates intractable Bayesian integrals using variational methods.
result Comparison of various approximation methods in literature.
The paper simplifies Bayesian deep learning by reducing high-dimensional parameter space to lower dimensions.
problem Scaling Bayesian inference to deep neural networks is challenging due to high dimensionality.
method Construct low-dimensional subspaces using SGD trajectories and apply elliptical slice sampling and variational inference.
result Bayesian model averaging over induced posterior in subspaces produces accurate predictions and well-calibrated uncertainty.
BLISS detects and separates astronomical sources quickly and accurately.
problem Detecting and separating overlapping astronomical sources in large images.
method Bayesian Light Source Separator (BLISS) using deep generative models and variational inference.
result BLISS can process megapixel images in seconds and produce highly accurate catalogs.
Paper develops efficient variational inference for sparse deep learning with theoretical guarantees.
problem Sparse deep learning's challenge of huge storage consumption and sparse structure recovery.
method Bayesian treatment with spike-and-slab priors and continuous relaxation of Bernoulli distribution for computationally efficient variational inferences.
result Provides variational posterior contraction rate, justifying consistency of the proposed method.
Global inducing points improve Bayesian neural network performance.
problem Improving Bayesian neural network performance.
method Adapting correlated approximate posterior to all layers in a Bayesian neural network and deep Gaussian processes using learned global inducing points.
result State-of-the-art performance on CIFAR-10 (86.7%) without data augmentation or tempering.
A new upper bound for variational inference improves the efficiency of Bayesian deep learning.
problem Improving variational inference in Bayesian deep learning.
method Presented a new upper bound (EUBO) for evidence, derived from KL-divergence and log marginal likelihood, and used SGD for optimization.
result The new upper bound (EUBO) is tighter than previous methods and outperforms state-of-the-art results in Bayesian neural networks.
Paper shows variational inference works well for sparse deep learning models.
problem Generalization of sparse deep learning models using variational inference.
method Theoretical analysis linking variational inference to Bayesian and nonparametric regression.
result Near-minimax rates of convergence for Hölder smooth functions in sparse deep learning.
New bounds enable training of probabilistic models for deep networks.
problem Training scalable latent variable models for deep networks.
method Introducing new variational bounds for specific output layers of neural networks.
result Analytical bounds for certain output layers allow training without re-parameterization or Monte Carlo approximations.
A new variational method improves deep neural network inference.
problem Overparametrized deep neural networks struggle with variational approximations.
method A novel variational family with two independent linear subspaces.
result State-of-the-art performance across various tasks and datasets.
Bayesian Deep Learning tackles inverse problems with neural networks and approximate computations.
problem Solving inverse problems with indirect measurements and uncertainties.
method Bayesian Deep Learning, using neural networks and approximate computations.
result Effective solutions for inverse problems using Bayesian Deep Learning.
We present a novel approach for training deep neural networks in a Bayesian way. Classical, i.e. non-Bayesian, deep learning has two major drawbacks both originating from the fact that network parameters are considered to be deterministic. First, model uncertainty cannot be measured thus limiting the use of deep learni…
Bayesian deep learning uses function-space priors to improve model uncertainty and robustness.
problem Bayesian deep learning struggles with model-specific weight-space priors that are hard to interpret and specify.
method Apply a Dirichlet prior in predictive space and perform approximate function-space variational inference.
result The approach improves uncertainty quantification, scalability, and adversarial robustness in large-scale image classification.
We develop a scalable deep non-parametric generative model by augmenting deep Gaussian processes with a recognition model. Inference is performed in a novel scalable variational framework where the variational posterior distributions are reparametrized through a multilayer perceptron. The key aspect of this reformulati…
Bayesian deep learning with heavy-tailed weights achieves near-optimal performance.
problem Deep neural networks with heavy-tailed weights achieve near-optimal performance in various contexts.
method Introduced a Bayesian deep learning prior based on heavy-tailed weights and ReLU activation, showing near-optimal minimax contraction rates.
result Posterior distribution achieves near-optimal minimax contraction rates, adaptive to smoothness and intrinsic dimension.
Trained neural networks perform Bayesian reasoning for tasks beyond their initial scope.
problem Performing Bayesian reasoning for tasks outside the trained neural networks' initial scope.
method Used deep generative models as priors and classification/regression networks as constraints. Approximated Bayesian inference through variational or sampling techniques.
result The approach built on top of already trained networks, expanding the addressable questions.
Improved phylogenetic inference with normalizing flows.
problem Limitations of current diagonal Lognormal branch length approximation in VBPI.
method Proposes VBPI-NF using normalizing flows to handle non-Euclidean branch length space.
result Significantly improves phylogenetic posterior estimation on real data.
Gradient-free method improves predictive accuracy for probabilistic models.
problem Balancing computational efficiency and robust predictive performance in deep learning.
method CAVI-CMN, a gradient-free variational method for conditional mixture networks.
result CAVI-CMN achieves competitive and often superior predictive accuracy compared to MLE with backpropagation.
Robustness to outliers is a central issue in real-world machine learning applications. While replacing a model to a heavy-tailed one (e.g., from Gaussian to Student-t) is a standard approach for robustification, it can only be applied to simple models. In this paper, based on Zellner's optimization and variational form…
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bay…
Improved variational approximation for deep Wishart process models.
problem Improving predictive performance of deep Wishart process models.
method Generalizing the Bartlett decomposition of the Wishart distribution to allow linear combinations of rows and columns.
result Better predictive performance achieved with minimal additional computation cost.
Amortized VI for DGPs learns efficient inference.
problem Expressive limitations in GP approximations.
method Amortized variational inference for DGPs.
result Improved expressive prior and posterior for DGPs.
GWI combines deep neural networks with Gaussian processes for better predictive performance and uncertainty quantification.
problem Combining deep learning with Gaussian process uncertainty quantification.
method Gaussian Wasserstein inference (GWI) using Wasserstein distance between Gaussian measures.
result GWI achieves state-of-the-art performance on benchmark datasets.
Probabilistic deep learning uses neural networks and models to handle uncertainty.
problem Handling uncertainty in deep learning models.
method Two approaches: probabilistic neural networks and deep probabilistic models.
result TensorFlow Probability library supports both approaches.
Stabilizes deep Bayesian neural networks with self-stabilizing priors.
problem Brittleness and difficulty in training deep Bayesian neural networks.
method Signal propagation theory, reformulated ELBO, self-stabilizing priors.
result Improved convergence and robustness in training deeper networks and noisier settings.
Study analyzes EU in variational inference for Bayesian deep learning.
problem Analyzing epistemic uncertainty in variational inference for Bayesian deep learning.
method Theoretical analysis and derivation of relations between generalization error and EU measurements. Proposing a new objective function for VI.
result Proposed new objective function significantly improves EU evaluation over existing VI methods.
Improved neural network ensembles using Stein Variational Newton updates.
problem Lack of efficient second-order information in current ensemble methods.
method Proposes a novel approximate Bayesian inference method integrating Stein Variational Newton updates with scalable Hessian approximations.
result Significantly faster convergence and more accurate posterior distribution approximations.
Compact parameterization improves Bayesian neural network performance.
problem Improving performance of Bayesian neural networks using variational methods.
method Restricting variational distribution to a k-tied Normal distribution with low-rank factorization.
result Compact parameterization improves signal-to-noise ratio and convergence speed.
Paper proposes new Bayesian neural network models for efficient learning.
problem Efficient learning and model compression in deep neural networks.
method Proposes Spike-and-Slab Group Lasso (SS-GL) and Spike-and-Slab Group Horseshoe (SS-GHS) priors for structured sparsity in Bayesian neural networks.
result Establishes competitive performance in prediction accuracy, model compression, and inference latency compared to baseline models.