Bayesian deep learning uses function-space priors to improve model uncertainty and robustness.
problem Bayesian deep learning struggles with model-specific weight-space priors that are hard to interpret and specify.
method Apply a Dirichlet prior in predictive space and perform approximate function-space variational inference.
result The approach improves uncertainty quantification, scalability, and adversarial robustness in large-scale image classification.
Hi-fi priors enhance BNNs by learning flexible activations.
problem Challenging to impose function-space priors on BNNs.
method Optimization techniques to learn flexible activations.
result BNNs with flexible activations can achieve desired priors.
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
Flow Annealing Posterior Sampling unifies stochastic-process regression and PDE inverse problems.
problem Function-space posterior sampling for stochastic processes and inverse problems.
method Flow Annealing Posterior Sampling (FAPS) using pretrained function-space flow-matching priors.
result Coherent posterior samples with accurate uncertainty quantification.
A new method converts neural networks to function space for scalable sequential learning.
problem Challenges in gradient-based deep learning for sequential data.
method Dual parameterization of neural networks from weight to function space.
result Efficient scaling, knowledge retention, and new data incorporation.
FTIP uses normalizing flows to improve posterior inference in function space.
problem Challenges in posterior inference with implicit-process priors.
method FTIP uses normalizing flows to define a richer variational distribution over combination weights.
result FTIP captures asymmetric and multimodal posterior structure better than Gaussian coefficient approximations.
Paper addresses variational inference issues in Bayesian neural networks.
problem Negative infinite ELBO for function-space priors in BNNs.
method Regularized KL divergence for well-defined function-space variational inference.
result Method provides competitive uncertainty estimates for BNNs.
A new machine learning method for Bayesian inverse problems in function spaces.
problem Bayesian inverse problems in function spaces with incompatibility of white noise sources.
method One-step generative transport with amortized neural operator and prior-aligned Gaussian random field.
result Generative operator trained on prior samples and noisy observations generates posterior samples efficiently.
We extend diffusion models to function spaces and introduce a new method for sampling from posterior distributions.
problem Sampling from posterior distributions in infinite-dimensional function spaces using diffusion models.
method Infinite-dimensional extension of Doob's h-transform, Supervised Guidance Training for efficient sampling. result We prove that diffusion models can be conditioned to sample from posterior distributions and introduce a simulation-free score matching objective.
Probabilistic neural networks are typically modeled with independent weight priors, which do not capture weight correlations in the prior and do not provide a parsimonious interface to express properties in function space. A desirable class of priors would represent weights compactly, capture correlations between weigh…
F-PACOH improves meta-learners' reliability in uncertain regions.
problem Overconfident uncertainty estimates in meta-learning.
method Meta-learning priors as stochastic processes in function space, directly steering predictions towards high epistemic uncertainty.
result Significantly outperforms other meta-learners in Bayesian Optimization.
Unified framework for Bayesian PDE-constrained inversion using physics-informed neural networks.
problem Incorporating prior distributions in function space into Bayesian PINN-based inversion.
method Functional-prior-based approaches (fpBPINN) to Bayesian PDE-constrained inversion using physics-informed neural networks (PINNs). Two complementary approaches: FPI-BPINN and fParVI-PINN.
result Accurate estimation of posterior distributions in seismic traveltime tomography and Darcy-flow permeability inversion.
Develops a new method for functional regression that works with non-Gaussian data.
problem Limited models for regression in function spaces with Gaussian process priors.
method Introduces Neural Operator Flows (OpFlow) for non-Gaussian function spaces.
result OpFlow enables robust and accurate uncertainty quantification for functional regression.
Method solves Bayesian inverse problems in function space without assuming log-concavity.
problem Bayesian inverse problems in infinite-dimensional nonlinear settings.
method Score-based diffusion models as a prior, Langevin-type MCMC on function spaces.
result Provable convergence bound for posterior sampling, dependent on score approximation.
New method tunes prior IP to data for flexible predictive distributions.
problem Challenges in approximate inference for large models with high parameter dependencies.
method Inducing-point representation of prior IP to approximate posterior process.
result Scalable method that tunes prior IP to data and provides accurate non-Gaussian predictive distributions.
The paper proposes a method for better uncertainty estimation in neural networks.
problem Estimating predictive uncertainty in neural networks is crucial but challenging.
method The paper proposes a function-space variational inference method to infer a posterior distribution over functions.
result The proposed method leads to state-of-the-art uncertainty estimation and predictive performance.
This work extends Gaussian process priors to neural operators for function space mappings.
problem Improving uncertainty quantification in deep neural networks.
method Extending Gaussian process priors to neural operators with conditions for convergence and computation of covariance functions.
result Arbitrary-depth neural operators with Gaussian kernels converge to function-valued GPs, enabling posterior computation in regression scenarios.
Bayesian neural networks fail at out-of-distribution detection, revealing fundamental issues.
problem Out-of-distribution detection with Bayesian neural networks.
method Study of Bayesian inference with function space priors and comparison to Gaussian processes.
result Bayesian inference with function space priors does not lead to good OOD detection.
Efficiently quantifies uncertainty in DeepONets for function spaces.
problem Uncertainty quantification in deep operator networks.
method Randomized prior ensembles for frequentist inference.
result Improved robustness and accuracy, reliable uncertainty estimates, out-of-distribution detection, and model bias quantification.
FunDiff models physical functions using diffusion and autoencoders.
problem Adapting generative models to continuous physical functions.
method Combines latent diffusion with function autoencoder, enforcing physical priors.
result Achieves optimal convergence rates for physical function estimation.
Bayesian Neural ODEs improve vessel trajectory prediction with better uncertainty estimates.
problem Challenges in predicting vessel trajectories from irregular AIS data.
method Adopted a Gaussian process (GP) kernel-based prior on the vector field evaluated at measurement points, combined with probabilistic multiple shooting for long trajectories.
result Improved accuracy and uncertainty quantification in vessel trajectory predictions.
This work explores function-space inference using KL divergence and proposes Bayesian linear regression as a benchmark.
problem Approximating the predictive posterior distribution of Bayesian models without parameter posterior approximation.
method Employing Kullback-Leibler divergence and proposing featurized Bayesian linear regression as a benchmark.
result Minimizing KL divergence leads to an ill-defined objective function, highlighting limitations of this approach.
New method calibrates neural network uncertainty for medical images.
problem Uncalibrated probabilistic outputs from deep neural networks in medical diagnosis.
method Functional space variational inference for Bayesian neural networks.
result Better calibrated uncertainty estimates at lower computational cost.
MARS meta-learns function scores for improved predictive accuracy and uncertainty.
problem Difficulty in specifying expressive priors for Bayesian meta-learning.
method Meta-learning the score function of data-generating process marginals in the function space.
result State-of-the-art predictive accuracy and improved uncertainty estimates.
Study connects Gaussian processes and regularization for sequence-function mappings.
problem Understanding and interpreting sequence-function maps in biology.
method Relates Gaussian process priors, regularization, and gauge fixing in overparameterized weight space.
result Established the relationship between regularized regression and Gaussian processes in function space.
New Gaussian priors for neural networks improve scalability and Bayesian inference stability.
problem Scalability and stability issues in Bayesian neural network inference.
method Introduces a new Gaussian neural network prior with decreasing variance in network width, enabling stable MCMC sampling.
result The new prior enables stable MCMC sampling for Bayesian neural network inference, improving scalability and stability.
Develops theory for conditional optimal transport in infinite-dimensional spaces.
problem Bayesian inference with functional parameters in infinite-dimensional spaces.
method Theory of constrained optimal transport for block-triangular maps.
result Regularity estimates on conditioning maps from prior to posterior.
The Brownian bridge serves as a physics-informed prior for solving the Poisson equation.
problem Reconstructing physical fields from limited and noisy data with known governing equations.
method Formalizing inverse problems via Bayesian inference in function spaces using a Brownian bridge Gaussian process.
result The Brownian bridge Gaussian process can be viewed as a physics-constrained prior for the Poisson equation, allowing for a fully Bayesian framework.
The study analyzes a three-layer neural network's training dynamics using a functional-space mean-field theory.
problem Understanding the training dynamics of partially-trained three-layer neural networks.
method Generalized mean-field theory to functional spaces, proving convergence and feature learning.
result The training loss of the model decays to zero at a linear rate in the L2 regression setting. New method encodes function preferences into neural nets for better generalization.
problem Challenges in encoding explicit function preferences in neural network training.
method Function-space empirical Bayes (FSEB) regularization.
result FSEB leads to near-perfect semantic shift detection and improved generalization.
Unified methodology for estimating optimal transport maps in various function spaces.
problem Estimating the function T given samples from P and T♯P. method Unified methodology based on Poincaré inequality and smooth convex function gradient.
result Nearly sharp results in various settings, including normal distribution and neural networks.
Study on Bayesian transformers finds issues with weight-space inference and prior specification.
problem Challenges in obtaining meaningful uncertainty estimates for transformer models.
method Proposed a novel method based on implicit reparameterization of the Dirichlet distribution for variational inference on attention weights.
result Proposed method performs competitively with baselines in estimating predictive uncertainty.
Bayesian neural network (BNN) priors are defined in parameter space, making it hard to encode prior knowledge expressed in function space. We formulate a prior that incorporates functional constraints about what the output can or cannot be in regions of the input space. Output-Constrained BNNs (OC-BNN) represent an int…
Gaussian processes are flexible function approximators, with inductive biases controlled by a covariance kernel. Learning the kernel is the key to representation learning and strong predictive performance. In this paper, we develop functional kernel learning (FKL) to directly infer functional posteriors over kernels. I…
Paper analyzes Langevin dynamics for solving infinite-dimensional Bayesian inverse problems.
problem Solving high-dimensional Bayesian inverse problems in infinite-dimensional function spaces.
method Preconditioned Langevin dynamics with score-based generative models (SGMs).
result Derives error estimates and sufficient conditions for global convergence in Kullback-Leibler divergence.
Bayesian neural networks with functional priors improve surrogate modeling in mechanics.
problem Challenges in integrating prior knowledge and quantifying uncertainties in high-dimensional NN parameter spaces.
method Anchored ensembling to integrate a priori information and learn low-rank correlations between NN parameters.
result Effective transfer of knowledge between function-space and parameter-space priors improves surrogate model accuracy and uncertainty estimation.
We consider a Gaussian process formulation of the multiple kernel learning problem. The goal is to select the convex combination of kernel matrices that best explains the data and by doing so improve the generalisation on unseen data. Sparsity in the kernel weights is obtained by adopting a hierarchical Bayesian approa…
K-priors enable quick adaptation with minimal retraining.
problem Machine learning models struggle to adapt to changes efficiently.
method Combines weight and function-space priors to reconstruct past gradients.
result Adaptation with K-priors achieves similar performance to full retraining with less data.
DVIP improves on IP-based methods by using IPs as priors over latent functions.
problem Limited expressiveness of IP-based models, especially in function space.
method Proposes DVIP, a multi-layer generalization of IPs, and scalable variational inference.
result DVIP outperforms previous IP-based methods and deep GPs in regression and classification tasks.
New insights into how neural networks learn features, especially when they are very wide.
problem Understanding how gradient flow in wide neural networks selects solutions, especially in the feature-learning regime.
method Axiomatizing the canonical regularizer as a function-space energy and lift, and deriving geodesic ridge for the feature-learning regime.
result Gradient flow in feature-learning networks biases towards ridge regularization, distorting the inductive bias and damaging pretrained networks.
New divergences improve estimation and GAN training performance.
problem Improving estimation and training in machine learning models.
method Function-space regularized Rényi divergences.
result New divergences reduce variance and improve training performance.
Develops methods to measure and set function-space learning rates in neural networks.
problem Measuring and optimizing changes in neural network output functions.
method Efficient methods to measure and set function-space learning rates, requiring minimal computational overhead.
result Demonstrates FLeRM (Function-space Learning Rate Matching) for hyperparameter transfer across model scales.
Better uncertainty estimates for neural networks using Gaussian process priors.
problem Poor uncertainty estimates in neural networks, especially on out-of-distribution data.
method Characterize the function-space prior of an ensemble of infinitely-wide neural networks as a Gaussian process and use it to build a probabilistic model.
result The approach improves calibration of neural networks, especially under distributional shift.
We propose a novel Bayesian approach to solve stochastic optimization problems that involve finding extrema of noisy, nonlinear functions. Previous work has focused on representing possible functions explicitly, which leads to a two-step procedure of first, doing inference over the function space and second, finding th…
Paper characterizes embeddability of function spaces into Lp-type RKBS via metric entropy.
problem Characterizing embeddability of function spaces into Lp-type RKBS. method Establishes a connection between metric entropy growth and embeddability.
result A bound on metric entropy growth allows embedding into Lp-type RKBS. A method for eliciting expert beliefs using preferential questions and normalizing flows.
problem Eliciting high-dimensional probability distributions from noisy judgments.
method Normalizing flows based on preferential questions with a novel functional prior.
result The method allows for the inference of arbitrarily flexible densities from preferential judgments.
New method uses trainable activations to make BNNs behave like GPs.
problem Making Bayesian Neural Networks (BNNs) behave like Gaussian Processes (GPs).
method Introduced trainable activations and periodic activations to map GP priors to BNNs. Used 2-Wasserstein distance for optimization.
result Method consistently outperforms existing approaches or matches heuristic methods.
Deep Gaussian Processes are reinterpreted as deep trigonometric networks for tractable inference.
problem Challenging inference in DGPs due to intractable marginalization in latent function space.
method Viewing DGPs as deep trigonometric networks with Bochner's theorem, and using the wide limit with a bottleneck to translate DGPs into deep trigonometric networks.
result The weight space view yields the same effective covariance functions as obtained in function space, and varying prior distributions over network parameters is equivalent to employing different kernels.