Bayesian methods enhance deep learning models by improving reliability and uncertainty.
problem Improving reliability and uncertainty awareness in deep learning models.
method Approximate Bayesian inference techniques, including SG-MCMC and VI, applied to deep learning models.
result Enhanced posterior inference for deep learning models, particularly in neural networks and generative models.
Review of priors in Bayesian deep learning models.
problem The importance of prior choices in Bayesian deep learning models.
method Overview of different priors and methods of learning priors from data.
result Motivate practitioners to think carefully about prior specification.
In this paper we introduce ZhuSuan, a python probabilistic programming library for Bayesian deep learning, which conjoins the complimentary advantages of Bayesian methods and deep learning. ZhuSuan is built upon Tensorflow. Unlike existing deep learning libraries, which are mainly designed for deterministic neural netw…
Bayesian Neural Networks help quantify uncertainty in deep learning predictions.
problem Uncertainty quantification in deep learning predictions.
method Bayesian statistics applied to neural networks.
result Design, implementation, training, and evaluation of Bayesian Neural Networks.
Bayesian interpretation of deep ensembles improves uncertainty quantification.
problem Improving uncertainty estimation in deep learning models.
method Viewing deep ensembles as an approximate Bayesian method and specifying corresponding assumptions.
result Improved approximation leads to larger epistemic uncertainty, potentially more reliable predictions.
The problem of state estimation for unobservable distribution systems is considered. A deep learning approach to Bayesian state estimation is proposed for real-time applications. The proposed technique consists of distribution learning of stochastic power injection, a Monte Carlo technique for the training of a deep ne…
Study improves theoretical understanding of Bayesian deep learning for classification tasks.
problem Theoretical gap in understanding Bayesian approaches in deep learning for classification.
method PAC-Bayes bounds techniques and Spike-and-Slab priors for sparse deep learning.
result Established non-asymptotic results for prediction error, achieving minimax optimal rates.
While perception tasks such as visual object recognition and text understanding play an important role in human intelligence, the subsequent tasks that involve inference, reasoning and planning require an even higher level of intelligence. The past few years have seen major advances in many perception tasks using deep …
Bayesian sparsification reduces deep neural network complexity.
problem Complexity of deep neural networks limits their performance.
method Combines Bayesian shrinkage priors with stochastic variational inference.
result Bayesian model reduction (BMR) is a more efficient alternative for pruning model weights.
A comprehensive artificial intelligence system needs to not only perceive the environment with different `senses' (e.g., seeing and hearing) but also infer the world's conditional (or even causal) relations and corresponding uncertainty. The past decade has seen major advances in many perception tasks such as visual ob…
One of the main challenges of deep learning tools is their inability to capture model uncertainty. While Bayesian deep learning can be used to tackle the problem, Bayesian neural networks often require more time and computational power to train than deterministic networks. Our work explores whether fully Bayesian netwo…
Bayesian deep learning improves deep learning's capabilities across diverse settings.
problem Overlooked metrics, tasks, and data types in deep learning.
method Revisits strengths of Bayesian deep learning and addresses challenges.
result Bayesian deep learning can elevate deep learning's capabilities across diverse settings.
Bayesian Deep Learning tackles inverse problems with neural networks and approximate computations.
problem Solving inverse problems with indirect measurements and uncertainties.
method Bayesian Deep Learning, using neural networks and approximate computations.
result Effective solutions for inverse problems using Bayesian Deep Learning.
Combining Bayesian deep learning and split conformal prediction affects out-of-distribution coverage.
problem Improving out-of-distribution coverage in multiclass image classification.
method Combining Bayesian deep learning with split conformal prediction methods.
result Combining methods can reduce out-of-distribution coverage in some cases.
URSABench benchmarks Bayesian methods for deep learning models.
problem Scalability issues in Bayesian inference for deep learning.
method Open-source benchmark suite for assessing approximate Bayesian inference methods.
result Initial results show promise for addressing uncertainty and robustness in deep learning.
sBayFDNN bridges deep learning and functional data analysis for complex, structured data.
problem Challenges in functional data analysis, especially for complex, continuously structured data.
method Sparse Bayesian functional deep neural network (sBayFDNN) that learns adaptive functional embeddings and interpretable region selection.
result First theoretical guarantees for a Bayesian deep functional model, ensuring reliability and statistical rigor.
Unified theory linking Bayesian and ensemble methods in deep learning.
problem Uncertainty quantification in deep learning.
method Reformulating optimisation as convex optimisation in probability measures, studying Wasserstein gradient flows.
result Unified theory explaining success of deep ensembles over variational inference.
Bayesian deep learning tackles uncertainty in high-dimensional systems.
problem Uncertainty quantification in high-dimensional stochastic partial differential equations.
method Bayesian neural network (BNN) and Hamiltonian Monte Carlo (HMC) for efficient sampling of posterior distributions.
result The method efficiently handles high-dimensional problems with almost independent computational cost.
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
Bayesian neural networks (BNNs) augment deep networks with uncertainty quantification by Bayesian treatment of the network weights. However, such models face the challenge of Bayesian inference in a high-dimensional and usually over-parameterized space. This paper investigates a new line of Bayesian deep learning by pe…
BayesDLL offers a PyTorch library for Bayesian deep learning with large models.
problem Bayesian inference for large-scale deep networks.
method Variational inference, MC-dropout, stochastic-gradient MCMC, Laplace approximation.
result BayesDLL can handle Vision Transformers and pre-trained model weights as priors.
Bayesian deep learning for graphs improves graph classification and prediction tasks.
problem Graph classification reproducibility issues and lack of uncertainty quantification.
method Developed a Bayesian Deep Learning framework for graph learning, considering discrete and continuous edge features.
result Produces unsupervised embeddings for graph classification tasks reaching state-of-the-art performance.
Bayesian neural networks simplified with input augmentation.
problem Uncertainty in deep learning models.
method Layer-wise input augmentation to induce uncertainty distributions.
result State-of-the-art performance in uncertainty representation.
GPflux simplifies deep Gaussian processes for Python.
problem Challenges in implementing deep Gaussian processes.
method Python library for Bayesian deep learning with DGPs.
result Efficient, modular, and extensible library for DGPs.
Bayesian Predictive Coding improves deep learning uncertainty quantification.
problem Limitations of maximum a posteriori and maximum likelihood estimates in predictive coding.
method Developed Bayesian Predictive Coding (BPC) that estimates a posterior distribution over network parameters.
result BPC offers comparable uncertainty quantification to existing methods in Bayesian deep learning and improves convergence properties.
New method improves uncertainty estimation in Bayesian deep learning models.
problem Underestimation of predictive uncertainty in Neural Linear Models (NLMs).
method Proposes a novel training method to capture useful predictive uncertainties and incorporate domain knowledge.
result Traditional training procedures for NLMs can drastically underestimate uncertainty in data-scarce regions.
Bayesian framework improves deep classifier reliability.
problem Overconfident models under dataset shift.
method Bayesian inference with out-of-distribution data augmentation.
result Reliable uncertainty estimates for deep classifiers.
Bayesian network learns data invariances without augmentation.
problem Learning invariances in neural networks without manual design.
method Bayesian approach infers weight-sharing schemes from data.
result Model outperforms non-invariant networks on specific tasks.
RETINA Benchmark evaluates Bayesian deep learning on diabetic retinopathy detection.
problem Reliable uncertainty quantification for deep learning models in medical applications.
method Design and evaluation of a real-world diabetic retinopathy dataset and tasks.
result Benchmarking of Bayesian deep learning methods on diabetic retinopathy detection tasks.
Learning in deep models using Bayesian methods has generated significant attention recently. This is largely because of the feasibility of modern Bayesian methods to yield scalable learning and inference, while maintaining a measure of uncertainty in the model parameters. Stochastic gradient MCMC algorithms (SG-MCMC) a…
Deep learning methods continue to have a decided impact on machine learning, both in theory and in practice. Statistical theoretical developments have been mostly concerned with approximability or rates of estimation when recovering infinite dimensional objects (curves or densities). Despite the impressive array of ava…
Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.
problem Understanding the generalization performance of deep ReLU neural networks.
method Analyzes Bayesian free energy in overparametrized deep ReLU neural networks.
result Bayesian free energy is bounded even in overparametrized deep ReLU networks.
This work simplifies Bayesian inference for neural networks by identifying influential parameter directions.
problem High computational complexity in Bayesian inference for neural networks due to high-dimensional parameter space.
method Constructing an active subspace of influential parameter directions to reduce dimensionality.
result Effective and scalable Bayesian inference achieved via reduced active subspace.
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch n…
INP accelerates stochastic simulations using deep Bayesian active learning.
problem Computational expense of stochastic simulations at fine-grained resolution.
method Interactive Neural Process (INP) framework combining spatiotemporal surrogate model and active learning acquisition function.
result STNP outperforms baselines in accelerating stochastic simulations and LIG achieves state-of-the-art for Bayesian active learning.
Bayesian learning rule unifies and generalizes various machine learning algorithms.
problem Machine learning algorithms are diverse and not always understood.
method Bayesian principles and natural gradients are used to derive algorithms.
result Derives a wide range of algorithms including classical and modern ones.
Bayesian deep learning method using subnetwork inference.
problem Improving deep neural networks' calibration and efficiency.
method Perform inference over a subset of model weights, keeping others as point estimates.
result Subnetwork inference enables accurate predictive posteriors without full network approximations.
Study on Bayesian deep linear networks with multiple outputs and convolutional layers.
problem Characterize feature learning in finite-width Bayesian deep linear networks.
method Exact and analytical formulas for joint and posterior distributions, using large deviation theory.
result Quantitative description of feature learning in infinite-width regime.
Bayesian neural networks tutorial via MCMC in Python.
problem Bayesian inference for parameter estimation and uncertainty quantification in deep learning models.
method MCMC sampling methods to implement Bayesian inference, including advanced proposal distributions.
result Challenges in sampling multi-modal posterior distributions for Bayesian neural networks.
We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitt…
Bayesian deep learning avoids underfitting by projecting onto null space of generalized Gauss-Newton matrix.
problem Bayesian deep learning often underfits, leading to less accurate predictions than point estimates.
method Proposes a matrix-free algorithm to project onto the null space of the generalized Gauss-Newton matrix, ensuring Bayesian predictions do not underfit.
result The method scales to large models, including vision transformers with 28 million parameters, and avoids underfitting.
Bayesian Scattering offers a simple baseline for image data uncertainty.
problem Lack of interpretable, mathematically grounded uncertainty quantification methods for image data.
method Coupling wavelet scattering transform with a simple probabilistic head.
result Bayesian Scattering provides sensible uncertainty estimates under distribution shifts.
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
Bayesian learning made scalable with posteriors library.
problem Computational challenges in Bayesian learning with modern models.
method Introducing posteriors library and tempered MCMC.
result Bayesian approximations are useful and scalable.
Study compares random and learned features in deep Bayesian linear models.
problem Understanding how feature learning affects generalization in deep learning.
method Comparing deep random feature models to deep networks with trained layers.
result Random feature models can display double-descent behavior, while deep networks do not.
Bayesian optimization uses BNNs as efficient surrogate models for expensive function evaluations.
problem Optimizing expensive objective functions using Gaussian process surrogates.
method Study of Bayesian neural networks (BNNs) as alternatives to standard Gaussian process (GP) surrogates for optimization.
result Infinite-width BNNs are particularly promising, especially in high dimensions.
Deep ensembles mimic Bayesian averaging with learned priors.
problem Quantifying uncertainty in neural networks.
method Showed deep ensembles perform exact Bayesian averaging with an implicitly learned data-dependent prior.
result Deep ensembles are Bayesian and provide an explanation for their strong empirical performance.
Improves Bayesian inference for deep models to better approximate posterior distributions.
problem Bayesian deep learning struggles with intractable posterior distributions, leading to overconfident predictions.
method Uses variational inference to approximate posterior distributions, proposing a unified view and improving inference for deep Gaussian processes.
result Variational inference can provide a lower bound for marginal likelihood, facilitating model selection and optimization.