Bayesian Neural Networks help quantify uncertainty in deep learning predictions.
problem Uncertainty quantification in deep learning predictions.
method Bayesian statistics applied to neural networks.
result Design, implementation, training, and evaluation of Bayesian Neural Networks.
Bayesian methods enhance deep learning models by improving reliability and uncertainty.
problem Improving reliability and uncertainty awareness in deep learning models.
method Approximate Bayesian inference techniques, including SG-MCMC and VI, applied to deep learning models.
result Enhanced posterior inference for deep learning models, particularly in neural networks and generative models.
In this paper we introduce ZhuSuan, a python probabilistic programming library for Bayesian deep learning, which conjoins the complimentary advantages of Bayesian methods and deep learning. ZhuSuan is built upon Tensorflow. Unlike existing deep learning libraries, which are mainly designed for deterministic neural netw…
Method selects the best deep learner for time-series prediction using Bayesian networks.
problem Selecting the most effective deep learning model for time-series prediction.
method Bayesian network selects deep learners based on input variables and cluster training data.
result Threshold value determines which deep learners predict time-series data robustly.
One of the main challenges of deep learning tools is their inability to capture model uncertainty. While Bayesian deep learning can be used to tackle the problem, Bayesian neural networks often require more time and computational power to train than deterministic networks. Our work explores whether fully Bayesian netwo…
The problem of state estimation for unobservable distribution systems is considered. A deep learning approach to Bayesian state estimation is proposed for real-time applications. The proposed technique consists of distribution learning of stochastic power injection, a Monte Carlo technique for the training of a deep ne…
Bayesian neural networks (BNNs) augment deep networks with uncertainty quantification by Bayesian treatment of the network weights. However, such models face the challenge of Bayesian inference in a high-dimensional and usually over-parameterized space. This paper investigates a new line of Bayesian deep learning by pe…
Review of priors in Bayesian deep learning models.
problem The importance of prior choices in Bayesian deep learning models.
method Overview of different priors and methods of learning priors from data.
result Motivate practitioners to think carefully about prior specification.
BayesDLL offers a PyTorch library for Bayesian deep learning with large models.
problem Bayesian inference for large-scale deep networks.
method Variational inference, MC-dropout, stochastic-gradient MCMC, Laplace approximation.
result BayesDLL can handle Vision Transformers and pre-trained model weights as priors.
We present a new method to approximate posterior probabilities of Bayesian Network using Deep Neural Network. Experiment results on several public Bayesian Network datasets shows that Deep Neural Network is capable of learning joint probability distri- bution of Bayesian Network by learning from a few observation and p…
Study how depth affects inference in deep Bayesian neural networks.
problem Understanding how depth impacts inference in overparameterized linear Bayesian neural networks.
method Interpreting finite deep linear Bayesian neural networks as scale mixtures of Gaussian process predictors.
result Advances analytical understanding of how depth affects inference in a simple class of Bayesian neural networks.
Bayesian neural networks simplified with input augmentation.
problem Uncertainty in deep learning models.
method Layer-wise input augmentation to induce uncertainty distributions.
result State-of-the-art performance in uncertainty representation.
Bayesian network learns data invariances without augmentation.
problem Learning invariances in neural networks without manual design.
method Bayesian approach infers weight-sharing schemes from data.
result Model outperforms non-invariant networks on specific tasks.
Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.
problem Understanding the generalization performance of deep ReLU neural networks.
method Analyzes Bayesian free energy in overparametrized deep ReLU neural networks.
result Bayesian free energy is bounded even in overparametrized deep ReLU networks.
Bayesian sparsification reduces deep neural network complexity.
problem Complexity of deep neural networks limits their performance.
method Combines Bayesian shrinkage priors with stochastic variational inference.
result Bayesian model reduction (BMR) is a more efficient alternative for pruning model weights.
Deep neural networks (DNN) are versatile parametric models utilised successfully in a diverse number of tasks and domains. However, they have limitations---particularly from their lack of robustness and over-sensitivity to out of distribution samples. Bayesian Neural Networks, due to their formulation under the Bayesia…
Bayesian Deep Learning tackles inverse problems with neural networks and approximate computations.
problem Solving inverse problems with indirect measurements and uncertainties.
method Bayesian Deep Learning, using neural networks and approximate computations.
result Effective solutions for inverse problems using Bayesian Deep Learning.
This paper connects RND, deep ensembles, and Bayesian inference, providing a unified theoretical perspective.
problem Uncertainty quantification in deep learning models.
method Analysis of Random Network Distillation (RND) within the neural tangent kernel framework.
result The uncertainty signal from RND is equivalent to the predictive variance of a deep ensemble and can be made to mirror the centered posterior predictive distribution of Bayesian inference.
This work simplifies Bayesian inference for neural networks by identifying influential parameter directions.
problem High computational complexity in Bayesian inference for neural networks due to high-dimensional parameter space.
method Constructing an active subspace of influential parameter directions to reduce dimensionality.
result Effective and scalable Bayesian inference achieved via reduced active subspace.
We present a comprehensive study of multilayer neural networks with binary activation, relying on the PAC-Bayesian theory. Our contributions are twofold: (i) we develop an end-to-end framework to train a binary activated deep neural network, (ii) we provide nonvacuous PAC-Bayesian generalization bounds for binary activ…
This paper distills Bayesian posterior expectations for deep neural networks.
problem Improving deep neural network performance and uncertainty quantification.
method Develops a framework for distilling expectations from Bayesian posterior distributions using Monte Carlo samples.
result The framework successfully distills posterior predictive distribution and expected entropy.
Bayesian neural networks tutorial via MCMC in Python.
problem Bayesian inference for parameter estimation and uncertainty quantification in deep learning models.
method MCMC sampling methods to implement Bayesian inference, including advanced proposal distributions.
result Challenges in sampling multi-modal posterior distributions for Bayesian neural networks.
Bayesian deep learning tackles uncertainty in high-dimensional systems.
problem Uncertainty quantification in high-dimensional stochastic partial differential equations.
method Bayesian neural network (BNN) and Hamiltonian Monte Carlo (HMC) for efficient sampling of posterior distributions.
result The method efficiently handles high-dimensional problems with almost independent computational cost.
The key distinguishing property of a Bayesian approach is marginalization instead of optimization, not the prior, or Bayes rule. Bayesian inference is especially compelling for deep neural networks. (1) Neural networks are typically underspecified by the data, and can represent many different but high performing models…
Study on Bayesian deep linear networks with multiple outputs and convolutional layers.
problem Characterize feature learning in finite-width Bayesian deep linear networks.
method Exact and analytical formulas for joint and posterior distributions, using large deviation theory.
result Quantitative description of feature learning in infinite-width regime.
Global inducing points improve Bayesian neural network performance.
problem Improving Bayesian neural network performance.
method Adapting correlated approximate posterior to all layers in a Bayesian neural network and deep Gaussian processes using learned global inducing points.
result State-of-the-art performance on CIFAR-10 (86.7%) without data augmentation or tempering.
Generalization is essential for deep learning. In contrast to previous works claiming that Deep Neural Networks (DNNs) have an implicit regularization implemented by the stochastic gradient descent, we demonstrate explicitly Bayesian regularizations in a specific category of DNNs, i.e., Convolutional Neural Networks (C…
Bayesian sparsification improves complex-valued neural networks by 50-100x with minimal performance loss.
problem Efficiently compressing complex-valued neural networks for embedded systems.
method Extending Sparse Variational Dropout to complex-valued networks and conducting a numerical study.
result Achieved state-of-the-art performance on MusicNet with 50-100x compression.
Bayesian inference with deep, weakly nonlinear networks is solved rigorously.
problem Bayesian inference with neural networks of specific structure.
method Perturbative analysis of fully connected neural networks with a shaped nonlinearity.
result Neural network Bayesian inference can be equivalent to kernel methods under certain conditions.
PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.
problem Theoretical limits of fully connected deep neural networks with Gaussian priors.
method PAC-Bayesian bounds for fully connected Bayesian DNNs with Gaussian priors.
result PAC-Bayesian bounds match minimax-optimal rates in Besov space for nonparametric regression and binary classification.
Bayesian inference for wide neural networks using Edgeworth expansion.
problem Analyzing the non-Gaussian behavior of wide neural networks in Bayesian inference.
method Proposed a non-Gaussian distribution using multivariate Edgeworth expansion for finite-width neural networks.
result Derived non-Gaussian posterior distribution in Bayesian regression tasks.
During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample effic…
Paper proposes new Bayesian neural network models for efficient learning.
problem Efficient learning and model compression in deep neural networks.
method Proposes Spike-and-Slab Group Lasso (SS-GL) and Spike-and-Slab Group Horseshoe (SS-GHS) priors for structured sparsity in Bayesian neural networks.
result Establishes competitive performance in prediction accuracy, model compression, and inference latency compared to baseline models.
Bayesian Entropy Neural Networks enforce constraints on deep learning predictions.
problem Deep learning models lack well-defined constraints in their outputs.
method Bayesian Entropy Neural Networks (BENN) using Maximum Entropy principles and the method of multipliers.
result BENN improves model robustness and reliability across various applications.
We present a novel approach for training deep neural networks in a Bayesian way. Classical, i.e. non-Bayesian, deep learning has two major drawbacks both originating from the fact that network parameters are considered to be deterministic. First, model uncertainty cannot be measured thus limiting the use of deep learni…
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
Probabilistic deep learning uses neural networks and models to handle uncertainty.
problem Handling uncertainty in deep learning models.
method Two approaches: probabilistic neural networks and deep probabilistic models.
result TensorFlow Probability library supports both approaches.
PAC-Bayesian method improves deep learning generalization in medical imaging.
problem Overfitting in deep networks for medical imaging datasets.
method PAC-Bayesian framework applied to large (stochastic) networks.
result PAC-Bayesian bounds are competitive and more explainable than simpler methods.
Bayesian inference for deep neural networks using trace-class priors and MLMC.
problem Efficient Bayesian inference for deep neural networks.
method Trace-class neural network priors and Multilevel Monte Carlo method.
result Optimal computational complexity for Bayesian inference of TNN models.
Bayesian neural networks improve deep learning's accuracy and uncertainty estimation.
problem Overconfident predictions, adversarial attacks, and variability underestimation in deep models.
method Stochastic relaxation of feed-forward rectified neural networks with sparsity-promoting priors and Polya-Gamma data augmentation.
result Improved scalability and robustness to architectural design through approximate variational inference.
Deep networks become equivalent to linear models in large data regimes.
problem Understanding the behavior of deep neural networks in large data regimes.
method Information-theoretic analysis of fully-trained neural networks in proportional scaling regime.
result Proves deep Gaussian equivalence principle, showing deep networks can be simplified to linear models.
Bayesian methods improve OoD detection in deep networks.
problem Detecting Out-of-Distribution (OoD) inputs in deep neural networks.
method Three Bayesian inference approaches applied to VAE weights.
result Improved OoD detection scores over benchmarks.
Study compares random and learned features in deep Bayesian linear models.
problem Understanding how feature learning affects generalization in deep learning.
method Comparing deep random feature models to deep networks with trained layers.
result Random feature models can display double-descent behavior, while deep networks do not.
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
Bayesian deep learning improves out-of-distribution detection but not always.
problem Improving the reliability of deep learning models in uncertain or novel data.
method Validation of likelihood-based Bayesian models for out-of-distribution detection.
result Bayesian deep learning models can marginally outperform conventional neural networks in certain conditions.
New method learns priors for Bayesian neural networks from datasets.
problem Lack of prior beliefs in Bayesian deep learning.
method Amortised variational inference to learn priors from datasets.
result Flexible Bayesian neural networks for meta-learning and within-task minibatching.
Develops scalable Bayesian inference methods for neural networks.
problem Lack of model uncertainty in deep learning leading to overconfident predictions.
method Linearised Laplace approximation, conjugate Gaussian-linear models, stochastic gradient descent, sample-based EM algorithm.
result Equips neural networks with model uncertainty using scalable methods.
This work tackles the challenge of Bayesian deep learning by proposing a new framework for matching Gaussian process priors with neural network parameters.
problem The challenge of specifying priors over neural network parameters, which affects the induced functional prior and is uncontrolled.
method The approach involves defining functional priors using Gaussian processes and matching these priors with the functional prior of neural networks through the minimization of Wasserstein distance.
result The proposed framework offers systematic performance improvements over alternative priors and approximate Bayesian deep learning approaches.