QLA improves Bayesian uncertainty estimation for DNNs without increasing computational cost.
problem Overconfident out-of-distribution predictions from DNNs.
method Proposes Quadratic Laplace Approximation (QLA) to improve Bayesian uncertainty quantification.
result QLA yields modest yet consistent uncertainty estimation improvements over Linearized Laplace Approximation (LLA) on five regression datasets.
The paper addresses the invariance issue in Bayesian neural networks using linearized Laplace approximation.
problem Bayesian neural networks fail to maintain invariance under reparameterization, leading to different posterior densities for identical functions.
method Developed a geometric view of reparameterizations and a Riemannian diffusion process to extend reparameterization invariance to neural network predictive.
result Empirically improved posterior fit through approximate posterior sampling.
Bayesian deep learning method using subnetwork inference.
problem Improving deep neural networks' calibration and efficiency.
method Perform inference over a subset of model weights, keeping others as point estimates.
result Subnetwork inference enables accurate predictive posteriors without full network approximations.
New method corrects Laplace/BIC errors in singular models, revealing effective dimension.
problem Laplace/BIC errors in singular models due to incorrect effective dimension assumption.
method RLCT (real log canonical threshold) to correct effective dimension in linear models.
result Correct evidence slope and effective dimension estimation in linear settings.
A novel Laplace-approximated Bayesian Tensor Network Kernel Machine (LA-TNKM) provides principled uncertainty estimates.
problem How to provide principled uncertainty estimates for tensor network kernel machines.
method Employing a linearized Laplace approximation for Bayesian inference.
result Consistently matches or surpasses Gaussian Processes and BNNs across diverse UCI regression benchmarks.
Bayesian nonparametric models get better posterior estimates via SPDE methods.
problem Estimating posterior distributions in nonparametric Bayesian models.
method Extending diffusion methods to SPDEs on Hilbert spaces for posterior contraction and Laplace approximation.
result Derivation of posterior contraction rates and finite-sample Bernstein von Mises results.
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
Several numerical approximation strategies for the expectation-propagation algorithm are studied in the context of large-scale learning: the Laplace method, a faster variant of it, Gaussian quadrature, and a deterministic version of variational sampling (i.e., combining quadrature with variational approximation). Exper…
Paper develops federated GLMM algorithms for analyzing hierarchical data.
problem Analyzing hierarchical data with non-independent observations in a federated setting.
method Developed two federated GLMM algorithms using Laplace and Gaussian Hermite approximations.
result Federated GLMM can handle hierarchical data and achieve comparable or superior performance.
New method optimizes hyperparameters in deep learning models efficiently.
problem Manual hyperparameter tuning in deep learning models is inefficient and requires expertise.
method Introduces lower bounds to the linearized Laplace approximation of the marginal likelihood using neural tangent kernels.
result Optimization of hyperparameters can be significantly accelerated using the method.
New methods improve Laplace approximations for deep neural networks by selecting key parameters.
problem Improving uncertainty quantification in deep neural networks using computationally feasible approximations.
method Gradient-Laplace and Greedy-Laplace methods for selecting parameters in sub-network Laplace approximations.
result Gradient-Laplace method outperforms existing heuristic approaches and provides formal optimality guarantees.
New method samples DPPs efficiently without downsampling or low-rank approximations.
problem Efficient sampling from DPPs for diverse selections.
method Directly approximates distribution function of linear statistics using Laplace inversion.
result Scalable sampling for general DPPs, beyond symmetric kernels.
Ribbon: Scalable Approximation and Robust Uncertainty Quantification
problem Reliably quantifying predictive uncertainty for complex models
method Ribbon, a scalable approximation to Dirichlet-reweighted bootstrap uncertainty
result Asymptotically equivalent to a flat-prior Laplace approximation under correct likelihood specification, recovers robust sandwich covariance under misspecification
A new method combines Laplace and Variational Bayes for scalable inference.
problem Complex models and large datasets make exact inference infeasible.
method Low-Rank Variational Bayes Correction (VBC) using Laplace method and Variational Bayes correction in a lower dimension.
result The method ensures scalability in both model complexity and data size.
Proposes a new method to approximate Bayesian predictive uncertainty.
problem Bayesian uncertainty quantification in model predictions.
method Self-supervised learning approach to approximate posterior predictive distribution.
result SSLA and ASSLA outperform classical Laplace approximations in predictive calibration.
A new method for efficient uncertainty estimation in deep learning models.
problem High computational costs in uncertainty estimation for deep neural networks.
method Variational sparse Gaussian Process approximation of the Linearized Laplace Approximation.
result Sub-linear training time and improved performance compared to existing methods.
Improves Laplace approximation for Bayesian inference on Riemannian manifolds.
problem Inaccurate Gaussian approximations for complex targets and finite-data posteriors.
method Develops alternative variants of the Laplace approximation using a Riemannian metric.
result Exact approximations at the limit of infinite data, improving practical performance.
LLA shows strong performance in Bayesian optimization but has unbounded search space issues.
problem Applying LLA in unbounded search spaces for Bayesian optimization.
method Linearized-Laplace approximation applied to Bayesian optimization problems.
result LLA demonstrates strong performance but also presents unbounded search space challenges.
Revisits online Laplace methods for neural networks, showing they are sound under certain conditions.
problem Online Laplace methods violate the Laplace approximation's critical assumption.
method Re-derives online Laplace methods, showing they target a variational bound on a mode-corrected variant of the Laplace evidence.
result Online Laplace and its mode-corrected counterpart share stationary points that satisfy the Laplace method's assumption.
Bayesian neural networks approximate Gaussian, this method adapts to non-Gaussian posteriors.
problem Bayesian neural networks struggle with non-Gaussian posteriors, leading to poor performance.
method Proposes a Riemannian Laplace approximation to adapt to the shape of the true posterior.
result Consistently improves over conventional Laplace approximation across tasks.
New iterative methods improve Vecchia-Laplace approximations for large data sets.
problem Inaccurate and slow Vecchia-Laplace approximations for large data sets.
method Iterative methods to improve Vecchia-Laplace approximations, including preconditioners and novel methods for predictive variances.
result Order of magnitude speed-up and threefold increase in prediction accuracy compared to state-of-the-art methods.
Enhances predictive performance in Bayesian deep learning via generalized Laplace approximation.
problem Inconsistency in Bayesian deep learning.
method Interprets posterior tempering as a correction for model misspecification and recalibration of priors. Introduces generalized Laplace approximation.
result Generalized Laplace approximation enhances predictive performance.
New metrics using Laplace approximation improve Gaussian process model selection.
problem Finding a balance between model accuracy, interpretability, and simplicity.
method Introducing multiple metrics based on the Laplace approximation to evaluate Gaussian process models.
result Our metrics provide comparable performance to dynamic nested sampling but are significantly faster.
We establish an explicit expression for the conditional Laplace transform of the integrated Volterra Wishart process in terms of a certain resolvent of the covariance function. The core ingredient is the derivation of the conditional Laplace transform of general Gaussian processes in terms of Fredholm's determinant and…
Fast approximate inference for non-Gaussian data.
problem Efficient inference for non-Gaussian data.
method Laplace Matching for fast approximate inference in latent Gaussian models.
result Achieves high approximation quality with low computational cost.
While learning the maximum likelihood value of parameters of an undirected graphical model is hard, modelling the posterior distribution over parameters given data is harder. Yet, undirected models are ubiquitous in computer vision and text modelling (e.g. conditional random fields). But where Bayesian approaches for d…
Geometrically reformulates the Laplace method for optimal transport.
problem Approximating integrals using the Laplace method without geometric interpretation.
method Introduces the Kim-McCann Riemannian metric to give a geometric formulation of the Laplace method.
result Expresses the first-order term of the Laplace method using geometric objects.
New algorithms improve likelihood of finding global optima in Bayesian inference.
problem Finding global optima in Bayesian inference is difficult due to nonconvexity.
method Developed two algorithms: consistent Laplace approximation (CLA) and consistent stochastic variational inference (CSVI).
result Both CSVI and CLA improve likelihood of obtaining global optima compared to standard methods.
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the we…
Proposes sampling from reverse diffusion posteriors for contextual bandits.
problem Complex distributions in contextual bandits.
method Approximate posterior sampling with a diffusion model prior using Laplace approximation.
result Empirically consistent and efficient approximations for contextual bandits.
We show that eigenvalues and eigenfunctions of the Laplace-Beltrami operator on a Riemannian manifold are approximated by eigenvalues and eigenvectors of a (suitably weighted) graph Laplace operator of a proximity graph on an epsilon-net.
New CRM models for sparse networks with linear edge growth.
problem Modeling extremely sparse networks with tractable properties.
method Introduced a new class of CRMs with index of variation α∈(0,1] based on mixtures of stable or generalized gamma processes.
result Models produce networks with near-linear edge growth, aligning with empirical evidence.
Develops scalable Bayesian inference methods for neural networks.
problem Lack of model uncertainty in deep learning leading to overconfident predictions.
method Linearised Laplace approximation, conjugate Gaussian-linear models, stochastic gradient descent, sample-based EM algorithm.
result Equips neural networks with model uncertainty using scalable methods.
MCMC complexity matches optimization for large n and d.
problem Lack of theoretical understanding of MCMC complexity for large n and d. method Comparison of MCMC, LA, and VI complexities for linear, logistic, and Poisson regression.
result MCMC complexity matches optimization complexity for n≳d. Variational Laplace improves Bayesian neural networks performance.
problem Improving Bayesian neural networks performance.
method Develops variational Laplace for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms other inference methods.
Combines Laplace approximations of deep networks for better uncertainty quantification.
problem Overconfident predictions on outliers in deep learning models.
method Gaussian mixture model posterior using weighted sum of Laplace approximations of pre-trained deep networks.
result Mitigates overconfidence 'far away' from training data.
Bayesian tensor train kernel machine uses Laplace approximation for scalable GP regression.
problem Scalability limitations of Gaussian process regression.
method Bayesian tensor train kernel machine with Laplace approximation and variational inference.
result VI replaces cross-validation and offers up to 65x faster training.
Variational Laplace improves Bayesian neural network performance without sampling.
problem Improving Bayesian neural network performance and calibration.
method Develops a new variational Laplace method for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms standard VI methods in test performance and calibration.
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences.…
LaLoRA prevents forgetting in LoRA fine-tuning.
problem Catastrophic forgetting in fine-tuned models.
method LaLoRA applies Laplace approximation to LoRA weights for regularization.
result Improved learning-forgetting trade-off with controllable regularization strength.
A new method for uncertainty estimation in neural networks using existing optimization steps.
problem Uncertainty quantification in deep neural networks.
method L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation.
result L2M method yields reasonable results without requiring changes in models or extra computational steps.
This purpose of this write-up is to share an idea for accurate computation of Laplace eigenvalues on a broad class of smooth domains. We represent the eigenfunction u as a linear combination of eigenfunctions corresponding to the common eigenvalue ρ2:\EQN{6}{1}{}{0}{\RD{\CELL{u(r,θ) =\sum_{n=0}^{N}P_{n}J_{n}(ρ) …
Logistic Gaussian process (LGP) priors provide a flexible alternative for modelling unknown densities. The smoothness properties of the density estimates can be controlled through the prior covariance structure of the LGP, but the challenge is the analytically intractable inference. In this paper, we present approximat…
The Laplace approximation calls for the computation of second derivatives at the likelihood maximum. When the maximum is found by the EM-algorithm, there is a convenient way to compute these derivatives. The likelihood gradient can be obtained from the EM-auxiliary, while the Hessian can be obtained from this gradient …
We compare various extensions of the Bradley-Terry model and a hierarchical Poisson log-linear model in terms of their performance in predicting the outcome of soccer matches (win, draw, or loss). The parameters of the Bradley-Terry extensions are estimated by maximizing the log-likelihood, or an appropriately penalize…
This paper improves Bayesian neural nets by using local linearization.
problem Underfitting in Bayesian neural networks.
method Local linearization of Bayesian neural networks to create a generalized linear model (GLM) for predictions.
result The GLM predictive resolves common underfitting problems of the Laplace approximation.
The future predictive performance of a Bayesian model can be estimated using Bayesian cross-validation. In this article, we consider Gaussian latent variable models where the integration over the latent values is approximated using the Laplace method or expectation propagation (EP). We study the properties of several B…
This work approximates full conformal prediction for neural networks without sample splitting.
problem Uncertainty quantification for neural network regression models.
method Approximating full conformal prediction using Gauss-Newton influence for post-hoc uncertainty estimation.
result Locally-adaptive and often tighter prediction intervals compared to split-CP.