Bayesian priors for neural networks are improved by incorporating weight correlations and tail behavior.
problem Improving Bayesian priors for neural networks to better reflect true beliefs and performance.
method Analyzed summary statistics of neural network weights in different architectures and incorporated these observations into new priors.
result Improved performance on image classification datasets by using new priors that account for weight correlations and tail behavior.
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
BNNpriors library improves Bayesian neural network inference with various prior distributions.
problem Challenges in choosing good prior distributions for Bayesian neural networks.
method State-of-the-art Markov Chain Monte Carlo inference with a wide range of predefined priors.
result Facilitates foundational discoveries on the nature of the cold posterior effect.
This work tackles the challenge of Bayesian deep learning by proposing a new framework for matching Gaussian process priors with neural network parameters.
problem The challenge of specifying priors over neural network parameters, which affects the induced functional prior and is uncontrolled.
method The approach involves defining functional priors using Gaussian processes and matching these priors with the functional prior of neural networks through the minimization of Wasserstein distance.
result The proposed framework offers systematic performance improvements over alternative priors and approximate Bayesian deep learning approaches.
Enhances neural networks with prior function values to improve accuracy.
problem Improving neural network accuracy in regions without training data.
method Develops a probabilistic approach to augment BNNs with prior function values.
result Predictions rely more on prior information in uncertain regions.
Bayesian neural networks use ridgelet prior for uncertainty quantification.
problem Combining strong predictive performance with uncertainty quantification in Bayesian neural networks.
method Proposes a ridgelet prior that approximates a Gaussian process covariance function in the output space of the network.
result Establishes universality property allowing Bayesian neural networks to approximate any Gaussian process.
Bayesian neural network achieves nearly optimal performance in Besov space.
problem Bayesian neural networks in Besov space.
method Spike-and-slab prior and shrinkage prior for posterior convergence rate.
result The posterior convergence rate is nearly minimax and adaptive to unknown smoothness.
New method learns priors for Bayesian neural networks from datasets.
problem Lack of prior beliefs in Bayesian deep learning.
method Amortised variational inference to learn priors from datasets.
result Flexible Bayesian neural networks for meta-learning and within-task minibatching.
Paper improves speech separation by using deep neural networks for more accurate density priors.
problem Improving the accuracy of source priors for independent vector analysis in speech separation.
method Estimating the derivative of speech density using deep neural networks to optimize performance indices.
result Neural network density priors outperform previous ones in convergence speed and SIR.
Unified framework for Bayesian PDE-constrained inversion using physics-informed neural networks.
problem Incorporating prior distributions in function space into Bayesian PINN-based inversion.
method Functional-prior-based approaches (fpBPINN) to Bayesian PDE-constrained inversion using physics-informed neural networks (PINNs). Two complementary approaches: FPI-BPINN and fParVI-PINN.
result Accurate estimation of posterior distributions in seismic traveltime tomography and Darcy-flow permeability inversion.
This work explores how overparametrization and priors affect Bayesian neural network posteriors.
problem Symmetries, non-identifiabilities, and weight-space priors fragment and inflate BNN posteriors.
method We study the interplay between overparametrization and priors in BNN posteriors, deriving key phenomena and validating through experiments.
result Overparametrization induces structured, prior-aligned weight posterior distributions.
Proposes new priors for neural networks to improve generalization and uncertainty.
problem Improving generalization and uncertainty estimation in neural networks.
method Exploits scalable and structured posteriors as priors with generalization guarantees.
result Improves generalization and uncertainty estimation with non-vacuous bounds.
Proposes DSM priors for Bayesian neural networks to improve interpretability and robustness.
problem Bayesian neural networks struggle with interpretability, overconfidence, and adversarial attacks.
method Introduces Dirichlet scale mixture (DSM) priors to address these issues.
result DSM priors lead to sparse networks, robustness against adversarial attacks, and competitive predictive performance.
The paper deals with learning probability distributions of observed data by artificial neural networks. We suggest a so-called gradient conjugate prior (GCP) update appropriate for neural networks, which is a modification of the classical Bayesian update for conjugate priors. We establish a connection between the gradi…
Bayesian priors improve neural network performance on weak signals.
problem Challenges in encoding domain knowledge for weak signals in neural networks.
method Proposed a new joint prior over local scale parameters for feature sparsity and signal-to-noise ratio, optimized with Stein gradient.
result Improved prediction accuracy on various datasets, including genetics applications with weak and sparse signals.
Bayesian weight priors improve neural network learning of identity relations.
problem Neural networks struggle to learn abstract and systematic relations, especially identity relations.
method Extended RBP approach using Bayesian weight priors as a regularization term.
result Bayesian weight priors lead to perfect generalization for identity relations and do not hinder standard neural network learning.
Bayesian neural networks incorporate domain knowledge through variational inference.
problem Specifying priors for Bayesian neural networks that capture domain knowledge is challenging.
method Proposes a framework for integrating domain knowledge into BNN priors through variational inference.
result BNNs with proposed domain knowledge priors outperform those with standard priors, achieving better predictive performance.
Proposes a new prior for complex models to improve prediction accuracy.
problem Difficulty in specifying priors for complex models like neural networks.
method Predictive complexity priors defined by comparing model predictions to a reference model, transferred to parameters via change of variables.
result Improves model predictions by reducing unintuitive effects of traditional priors.
Neural-g models mixtures of densities with flexibility and accuracy.
problem Accurately estimating prior densities in g-models. method Neural network with softmax output for valid probability densities.
result Neural-g captures various prior shapes including flat, heavy-tailed, and discontinuous.
Bayesian neural networks' performance varies with prior choice, affecting their ability to identify unknowns.
problem The impact of prior choice on Bayesian neural networks' ability to identify unknowns.
method Evaluation of different prior distributions on classification tasks using BNNs and NNs with Monte Carlo dropout.
result Prior choice significantly impacts BNNs' ability to identify unknowns, affecting true and false positive rates.
Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dime…
Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (DWP), that exploi…
Study shows prior Lipschitz continuity can improve adversarial robustness of Bayesian Neural Networks.
problem Improving adversarial robustness of Bayesian Neural Networks.
method Analysis of i.i.d., zero-mean Gaussian priors and posteriors approximated via mean-field variational inference.
result Adversarial robustness is sensitive to the prior variance.
New theory for BNNs with Gaussian priors achieves optimal posterior concentration rates.
problem Lack of theoretical results for BNNs with Gaussian priors.
method New approximation theory for non-sparse DNNs with bounded parameters.
result BNNs with non-sparse general priors can achieve near-minimax optimal posterior concentration rates.
New Gaussian priors for neural networks improve scalability and Bayesian inference stability.
problem Scalability and stability issues in Bayesian neural network inference.
method Introduces a new Gaussian neural network prior with decreasing variance in network width, enabling stable MCMC sampling.
result The new prior enables stable MCMC sampling for Bayesian neural network inference, improving scalability and stability.
Rényi Neural Processes replace KL divergence with Rényi divergence to improve NP performance.
problem Parameterization coupling in Neural Processes leads to prior misspecification.
method Propose Rényi Neural Processes (RNP) by replacing KL divergence with Rényi divergence.
result Significant performance improvements in real-world problems, including better log-likelihoods.
Paper addresses variational inference issues in Bayesian neural networks.
problem Negative infinite ELBO for function-space priors in BNNs.
method Regularized KL divergence for well-defined function-space variational inference.
result Method provides competitive uncertainty estimates for BNNs.
This paper introduces hierarchical Gaussian process priors for neural networks to capture weight correlations and inductive biases.
problem Capturing weight correlations and inductive biases in neural networks.
method Hierarchical Gaussian process priors with unit embeddings and input-dependent kernels.
result Hierarchical Gaussian process priors provide competitive predictive performance and desirable uncertainty estimates.
Review of priors in Bayesian deep learning models.
problem The importance of prior choices in Bayesian deep learning models.
method Overview of different priors and methods of learning priors from data.
result Motivate practitioners to think carefully about prior specification.
We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activat…
New priors improve Bayesian neural networks without cooling.
problem Bayesian neural networks underfit on clean datasets.
method Introduce DirClip and confidence priors to replace cooling.
result DirClip and confidence priors outperform cold posterior.
RP-WNO extends WNO with uncertainty quantification, useful for scientists and engineers.
problem Uncertainty in predictions of deep learning models.
method Randomized Prior Wavelet Neural Operator (RP-WNO) with uncertainty quantification module.
result RP-WNO effectively estimates uncertainty in predictions.
R2D2-Net improves Bayesian neural networks by preventing over-shrinkage of important weights.
problem Bayesian neural networks struggle with choosing appropriate priors, leading to over-shrinkage or poor predictive performance.
method Proposes R2D2-Net with an R^2-induced Dirichlet Decomposition prior and variational Gibbs inference algorithm.
result R2D2-Net effectively shrinks irrelevant coefficients while preventing key features from over-shrinkage.
Proposes a new model for community detection using neural priors.
problem Community detection in graphs with node attributes.
method Neural-prior stochastic block model with belief propagation and approximate message passing algorithm.
result Identifies phase transitions and algorithmically hard regions for community detection.
Deep neural networks as image priors have been recently introduced for problems such as denoising, super-resolution and inpainting with promising performance gains over hand-crafted image priors such as sparsity and low-rank. Unlike learned generative priors they do not require any training over large datasets. However…
Hybrid Bayesian neural networks use function uncertainty for probabilistic inference.
problem Uncertainty in neural network weights is hard to specify and interpret.
method Integrates probabilistic layers with standard deterministic layers for function uncertainty.
result Improves probabilistic inference by encoding function uncertainty.
New method uses unlabelled data to improve Bayesian Neural Networks.
problem Lack of ability to use unlabelled data in conventional Bayesian Neural Networks.
method Self-supervised Bayesian Neural Networks using contrastive pretraining and variational lower bound optimization.
result Prior predictive distributions capture problem semantics better and improve predictive performance.
Enhances neural networks with prior knowledge through a composite kernel.
problem Lack of effective methods to incorporate prior knowledge into neural networks.
method Integrates a composite kernel combining a neural network kernel and a GP kernel for modeling known properties.
result Demonstrates superior performance and flexibility of the Implicit Composite Kernel (ICK) on synthetic and real-world data.
New approach uses prior knowledge to improve neural network representations.
problem Difficulty in learning robust internal representations without a large training set.
method Incorporates prior information into a pre-trained reasoning module.
result Improves representation learning in diverse self-supervised learning settings.
Bayesian neural networks with Mercer priors for interpretable uncertainty quantification.
problem Uncertainty quantification in neural networks, especially for complex input-to-output mappings.
method Introducing Mercer priors for BNNs, which approximate a specified GP and are scalable.
result BNNs with Mercer priors can approximate the uncertainty of a specified GP, making them interpretable and scalable.
New approach to neural networks by incorporating observation noise and arbitrary prior means.
problem Misspecification on noisy data and limitations of NTK-GP equivalence.
method Introducing a regularizer for observation noise and proposing a shifted network for arbitrary prior means.
result Removes key obstacles to practical Gaussian process modeling in neural networks.
Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodo…
Hi-fi priors enhance BNNs by learning flexible activations.
problem Challenging to impose function-space priors on BNNs.
method Optimization techniques to learn flexible activations.
result BNNs with flexible activations can achieve desired priors.
Bayesian neural networks improve uncertainty calibration with DAP priors.
problem Improving predictive uncertainty in deep learning models outside training data.
method Distance-Aware Prior (DAP) calibration method to correct overconfidence.
result Demonstrated effectiveness in various classification and regression tasks.
Proposes scale mixture of NNGPs for more flexible stochastic processes.
problem Limited focus on broadening the class of stochastic processes from NNGPs.
method Scale mixture of NNGPs with scale priors on last-layer parameters.
result Turns neural networks into a richer class of stochastic processes.
We propose a new approach, called cooperative neural networks (CoNN), which uses a set of cooperatively trained neural networks to capture latent representations that exploit prior given independence structure. The model is more flexible than traditional graphical models based on exponential family distributions, but i…
The paper proposes a method for better uncertainty estimation in neural networks.
problem Estimating predictive uncertainty in neural networks is crucial but challenging.
method The paper proposes a function-space variational inference method to infer a posterior distribution over functions.
result The proposed method leads to state-of-the-art uncertainty estimation and predictive performance.
MARS meta-learns function scores for improved predictive accuracy and uncertainty.
problem Difficulty in specifying expressive priors for Bayesian meta-learning.
method Meta-learning the score function of data-generating process marginals in the function space.
result State-of-the-art predictive accuracy and improved uncertainty estimates.