Review of priors in Bayesian deep learning models.
problem The importance of prior choices in Bayesian deep learning models.
method Overview of different priors and methods of learning priors from data.
result Motivate practitioners to think carefully about prior specification.
Informative Bayesian priors are often difficult to elicit, and when this is the case, modelers usually turn to noninformative or objective priors. However, objective priors such as the Jeffreys and reference priors are not tractable to derive for many models of interest. We address this issue by proposing techniques fo…
Statsformer validates and adapts LLM-derived semantic priors for improved supervised learning.
problem Unreliable semantic priors from LLMs can degrade supervised learning performance.
method Adapts LLM-derived feature scores into a family of learner-specific prior-injection mechanisms, calibrating their influence using out-of-fold validation.
result Improves prediction performance by adaptively downweighting unreliable LLM priors, ensuring a guardrailed statistical learning system.
This paper uses reference priors to improve deep learning models with unlabeled and labeled data.
problem Improving deep learning models with limited labeled data and unlabeled data from the same or related tasks.
method Develops and applies generalizations of reference priors for deep networks to exploit unlabeled and labeled data.
result Demonstrates new semi-supervised learning and pretraining methods for transfer learning.
New meta-reinforcement learning method improves performance in finite-horizon MDPs.
problem Improving meta-reinforcement learning in finite-horizon MDPs with shared optimal action-value functions.
method Proposes MTSRL and MTSRL+ algorithms with learned priors and covariance, coupled with prior-alignment technique for meta-regret guarantees.
result Achieves meta-regret guarantees with learned priors and covariance, outperforming prior-independent RL and bandit-only meta-baselines.
In this paper the problem of learning appropriate bias for an environment of related tasks is examined from a Bayesian perspective. The environment of related tasks is shown to be naturally modelled by the concept of an {\em objective} prior distribution. Sampling from the objective prior corresponds to sampling differ…
Study proposes learning optimal priors from data for better Bayesian inference.
problem Challenges the use of noninformative uniform priors in Bayesian inference.
method Machine learning approach to learn optimal priors from data using a target function.
result Study models consistently outperformed baseline models in Wikipedia category classification.
GOAT improves attention mechanisms by learning better priors.
problem Standard attention mechanisms use a naive uniform prior, limiting flexibility and generalization.
method GOAT introduces a trainable, continuous prior that replaces the uniform assumption, maintaining compatibility with optimized kernels.
result GOAT avoids representational trade-offs and learns an extrapolatable prior that combines positional flexibility with length generalization.
Paper shows how meta-learning can reduce prior learning cost.
problem Learning the prior in meta-learning with fast rates.
method Examined Gibbs algorithm in meta-learning context.
result Bernstein's condition holds at meta level, reducing prior learning cost.
We study the problem of learning shared structure \emph{across} a sequence of dynamic pricing experiments for related products. We consider a practical formulation where the unknown demand parameters for each product come from an unknown distribution (prior) that is shared across products. We then propose a meta dynami…
A new method learns priors for Bayesian optimisation to improve performance.
problem Bayesian optimisation tasks often assume strong similarity, which is violated in many cases.
method Replace strong similarity assumption with shape similarity, learn priors for hyperparameters.
result PLeBO and prior transfer find good inputs in fewer evaluations.
A new method learns hierarchical EBM models with diffusion schemes.
problem Challenges in learning EBM models with multi-modal distributions.
method Proposes a diffusion probabilistic scheme to learn EBM models in hierarchical latent spaces.
result Demonstrates superior performance on various tasks with diffusion-learned EBM.
This work tackles the challenge of Bayesian deep learning by proposing a new framework for matching Gaussian process priors with neural network parameters.
problem The challenge of specifying priors over neural network parameters, which affects the induced functional prior and is uncontrolled.
method The approach involves defining functional priors using Gaussian processes and matching these priors with the functional prior of neural networks through the minimization of Wasserstein distance.
result The proposed framework offers systematic performance improvements over alternative priors and approximate Bayesian deep learning approaches.
Bayesian weight priors improve neural network learning of identity relations.
problem Neural networks struggle to learn abstract and systematic relations, especially identity relations.
method Extended RBP approach using Bayesian weight priors as a regularization term.
result Bayesian weight priors lead to perfect generalization for identity relations and do not hinder standard neural network learning.
Using powerful posterior distributions is a popular approach to achieving better variational inference. However, recent works showed that the aggregated posterior may fail to match unit Gaussian prior, thus learning the prior becomes an alternative way to improve the lower-bound. In this paper, for the first time in th…
Study characterizes training and test risks for MAP regression with Gaussian priors.
problem Understanding high-dimensional behavior of regularized linear regression with informative priors.
method Maximum a posteriori (MAP) regression with Gaussian priors, using random matrix theory.
result Closed-form risk formulas reveal the bias-variance-prior tradeoff and explain double descent.
Proposes new priors for neural networks to improve generalization and uncertainty.
problem Improving generalization and uncertainty estimation in neural networks.
method Exploits scalable and structured posteriors as priors with generalization guarantees.
result Improves generalization and uncertainty estimation with non-vacuous bounds.
New method learns priors for Bayesian neural networks from datasets.
problem Lack of prior beliefs in Bayesian deep learning.
method Amortised variational inference to learn priors from datasets.
result Flexible Bayesian neural networks for meta-learning and within-task minibatching.
MetaTS learns to explore better by meta-learning prior from bandit instances.
problem Efficient exploration in bandit problems.
method MetaThompson Sampling (MetaTS) that meta-learns the prior from bandit instances.
result MetaTS quickly adapts to unknown prior and achieves better exploration.
HyperBO+ pre-trains a universal prior for Bayesian optimization across different domains.
problem Bayesian optimization requires domain-specific priors, limiting its applicability.
method Two-step pre-training method for hierarchical Gaussian processes.
result HyperBO+ achieves lower regrets on unseen search spaces.
SAHMM-VAE separates sources adaptively using hidden Markov priors.
problem Unsupervised blind source separation.
method Source-wise adaptive Hidden Markov prior variational autoencoder.
result Different latent dimensions align with different source-specific temporal organizations.
This paper explores minimax-Bayes solutions for reinforcement learning problems.
problem How to select appropriate priors for decision making under uncertainty in sequential decision making.
method Study of minimax-Bayes solutions for various reinforcement learning problems.
result Minimax policies are more robust than standard priors.
In variational autoencoders, the prior on the latent codes z is often treated as an afterthought, but the prior shapes the kind of latent representation that the model learns. If the goal is to learn a representation that is interpretable and useful, then the prior should reflect the ways in which the high-level fact…
Bayesian algorithms perform well even with misspecified priors, especially in meta-learning.
problem Performance degradation of Bayesian algorithms with misspecified priors.
method Thompson sampling and meta-learning analysis with misspecified priors.
result Thompson sampling's performance degrades gracefully with misspecification, with a bound of ildeO(H2ε). New findings suggest latent regularization is unnecessary for high-quality image generation.
problem Improving image generation quality without latent regularization.
method Investigated the effect of latent regularization on image generation using learned priors.
result In the case of a sufficiently expressive prior, latent regularization is not necessary and may harm image quality.
Diffusion models help in learning priors for Thompson Sampling in bandit problems.
problem Learning effective strategies for diverse bandit tasks.
method Training a denoising diffusion model to learn task distributions, combining with Thompson Sampling.
result The approach significantly improves performance across different bandit tasks.
Hi-fi priors enhance BNNs by learning flexible activations.
problem Challenging to impose function-space priors on BNNs.
method Optimization techniques to learn flexible activations.
result BNNs with flexible activations can achieve desired priors.
Recently there has been an increased interest in unsupervised learning of disentangled representations using the Variational Autoencoder (VAE) framework. Most of the existing work has focused largely on modifying the variational cost function to achieve this goal. We first show that these modifications, e.g. beta-VAE, …
We introduce a model-based reconstruction framework with deep learned (DL) and smoothness regularization on manifolds (STORM) priors to recover free breathing and ungated (FBU) cardiac MRI from highly undersampled measurements. The DL priors enable us to exploit the local correlations, while the STORM prior enables us …
Learning the network structure underlying data is an important problem in machine learning. This paper introduces a novel prior to study the inference of scale-free networks, which are widely used to model social and biological networks. The prior not only favors a desirable global node degree distribution, but also ta…
Proposes a new prior for complex models to improve prediction accuracy.
problem Difficulty in specifying priors for complex models like neural networks.
method Predictive complexity priors defined by comparing model predictions to a reference model, transferred to parameters via change of variables.
result Improves model predictions by reducing unintuitive effects of traditional priors.
Most deep latent factor models choose simple priors for simplicity, tractability or not knowing what prior to use. Recent studies show that the choice of the prior may have a profound effect on the expressiveness of the model,especially when its generative network has limited capacity. In this paper, we propose to lear…
PRCD-MAP learns to trust imperfect priors in causal discovery, improving accuracy and robustness.
problem Tackles the brittle trade-off between blind trust and rejection of external priors in causal discovery.
method Proposes PRCD-MAP, a soft prior-consumption layer that assigns per-edge trust to imperfect priors and modulates regularization in a MAP objective.
result Enjoys a population-level safety guarantee and outperforms existing methods on real-world causal discovery tasks.
Adversarial meta-learning computes Gamma-minimax estimators for vague prior knowledge.
problem Estimating parameters with vague prior knowledge.
method Adversarial meta-learning algorithms for Gamma-minimax estimators.
result Convergence guarantees and neural network class for selection.
We propose to learn a hierarchical prior in the context of variational autoencoders to avoid the over-regularisation resulting from a standard normal prior distribution. To incentivise an informative latent representation of the data, we formulate the learning problem as a constrained optimisation problem by extending …
The paper highlights how machine learning calibrations can be biased by training data.
problem Machine learning calibrations can be biased by the training data, affecting downstream analyses.
method The paper examines simulation-based and data-based calibrations, highlighting their prior dependence and proposing solutions.
result A recently proposed Gaussian Ansatz approach can avoid some biases in simulation-based calibrations.
Bayesian method uses deep learning prior for CT reconstruction.
problem Imaging inverse problems in CT reconstruction.
method SA-Roundtrip prior with HMC-pCN sampler.
result Outperforms state-of-the-art methods in CT reconstruction.
SIGMA prior enables federated learning for non-factorizable models.
problem Current FL methods assume conditional independence, limiting applicability to non-factorizable models.
method SIGMA prior approximates deep generative model to induce conditional independence structure.
result SIGMA prior expands FL applicability to fields requiring modeling dependencies.
We study the robustness of active learning (AL) algorithms against prior misspecification: whether an algorithm achieves similar performance using a perturbed prior as compared to using the true prior. In both the average and worst cases of the maximum coverage setting, we prove that all α-approximate algorithms are …
Improves transparency and incorporates prior knowledge in Gaussian Process models.
problem Challenges in understanding and expressing prior assumptions in complex Bayesian models.
method Introduces self-explaining variational posterior distributions for Gaussian Processes.
result Allows incorporation of both general and feature-specific prior knowledge.
When fitting Bayesian machine learning models on scarce data, the main challenge is to obtain suitable prior knowledge and encode it into the model. Recent advances in meta-learning offer powerful methods for extracting such prior knowledge from data acquired in related tasks. When it comes to meta-learning in Gaussian…
New priors improve robustness and interpretability in penalized regression.
problem Improper priors in penalized regression lead to suboptimal solutions.
method Developed non-zero priors inspired by human decision heuristics.
result Robust priors yield excellent worst-case performance across various tasks.
Framework evaluates the impact of prior knowledge in deep learning models.
problem Mitigating data-driven model shortcomings like data dependence and generalization ability.
method Model-agnostic framework inspired by interpretable machine learning, assessing data volume and estimation range effects.
result Complex relationship between data and knowledge, including dependence, synergistic, and substitution effects.
We propose Learned Accept/Reject Sampling (LARS), a method for constructing richer priors using rejection sampling with a learned acceptance function. This work is motivated by recent analyses of the VAE objective, which pointed out that commonly used simple priors can lead to underfitting. As the distribution induced …
One of the most interesting features of Bayesian optimization for direct policy search is that it can leverage priors (e.g., from simulation or from previous tasks) to accelerate learning on a robot. In this paper, we are interested in situations for which several priors exist but we do not know in advance which one fi…
Study shows priors are crucial for accurate causal learning from unlabeled data.
problem Improving causal learning from unlabeled data.
method Investigated causal learning using Bayesian methods and analyzed the impact of priors.
result Factorized priors lead to factorized posteriors, aligning with independent causal mechanisms.
Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is suf…
Proposes a new method to learn meta-priors from data.
problem Improving learning systems with domain knowledge and controlling parameter learning rates.
method Hierarchical Empirical Bayes approach to decouple learning rates of features.
result Meta-prior learning improves performance and convergence time in various applications.