In this short note we provide an unbiased multilevel Monte Carlo estimator of the log marginal likelihood and discuss its application to variational Bayes.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “log marginal likelihood”
Improved Gaussian process regression with tighter log marginal likelihood bounds.
problem Improving predictive performance in Gaussian process regression models.
method Lower bound on log marginal likelihood using conjugate gradients.
result Improved predictive performance compared to other conjugate gradient based approaches.
SUMO provides unbiased log marginal likelihood estimation for latent variable models.
problem Biased estimates of log marginal likelihood in latent variable models.
method Randomized truncation of infinite series for unbiased estimation.
result Models trained with SUMO give better test-set likelihoods than standard methods.
Improves Gaussian process regression without bias.
problem Bias in Gaussian process regression estimates.
method Adaptive computation selection to minimize bias.
result Guaranteed small bias in log marginal likelihood estimates.
This work gives an in-depth derivation of the trainable evidence lower bound obtained from the marginal joint log-Likelihood with the goal of training a Multi-Modal Variational Autoencoder (MVAE).
In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through -fold partitioning or leave-$p…
Computing the marginal likelihood (ML) of a model requires marginalizing out all of the parameters and latent variables, a difficult high-dimensional summation or integration problem. To make matters worse, it is often hard to measure the accuracy of one's ML estimates. We present bidirectional Monte Carlo, a technique…
We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanatio…
Algorithms for Gaussian process, marginal likelihood methods or restricted maximum likelihood methods often require derivatives of log determinant terms. These log determinants are usually parametric with variance parameters of the underlying statistical models. This paper demonstrates that, when the underlying matrix …
A widely applicable Bayesian information criterion (Watanabe, 2013) is applicable for both regular and singular models in the model selection problem. This criterion tends to overestimate the log marginal likelihood. We identify an overestimating term of a widely applicable Bayesian information criterion. Adjustment of…
Evaluation of the marginal likelihood plays an important role in model selection problems. The widely applicable Bayesian information criterion (WBIC) and singular Bayesian information criterion (sBIC) give approximations to the log marginal likelihood, which can be applied to both regular and singular models. When the…
We propose a novel and flexible rank-breaking-then-composite-marginal-likelihood (RBCML) framework for learning random utility models (RUMs), which include the Plackett-Luce model. We characterize conditions for the objective function of RBCML to be strictly log-concave by proving that strict log-concavity is preserved…
Robust GP model detects and corrects sparse outliers.
problem Non-Gaussian noise in real-world data.
method Relevance pursuit for data-point-specific noise levels.
result Strong concavity and approximation guarantees for subset selection.
Training deep generative models with maximum likelihood remains a challenge. The typical workaround is to use variational inference (VI) and maximize a lower bound to the log marginal likelihood of the data. Variational auto-encoders (VAEs) adopt this approach. They further amortize the cost of inference by using a rec…
Improved GP decoder training with SAS approximations.
problem Training expensive Gaussian process decoders is challenging and computationally expensive.
method Developed a new stochastic estimate of log-marginal likelihood based on cross-validation.
result SAS-GP improves robustness and reduces computational cost compared to variational autoencoders.
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Prob…
Though with progress, model learning and performing posterior inference still remains a common challenge for using deep generative models, especially for handling discrete hidden variables. This paper is mainly concerned with algorithms for learning Helmholz machines, which is characterized by pairing the generative mo…
Gaussian latent tree models, or more generally, Gaussian latent forest models have Fisher-information matrices that become singular along interesting submodels, namely, models that correspond to subforests. For these singularities, we compute the real log-canonical thresholds (also known as stochastic complexities or l…
Thermodynamic integration (TI) for computing marginal likelihoods is based on an inverse annealing path from the prior to the posterior distribution. In many cases, the resulting estimator suffers from high variability, which particularly stems from the prior regime. When comparing complex models with differences in a …
We propose a novel training procedure for improving the performance of generative adversarial networks (GANs), especially to bidirectional GANs. First, we enforce that the empirical distribution of the inverse inference network matches the prior distribution, which favors the generator network reproducibility on the se…
We propose Subsampling MCMC, a Markov Chain Monte Carlo (MCMC) framework where the likelihood function for observations is estimated from a random subset of observations. We introduce a highly efficient unbiased estimator of the log-likelihood based on control variates, such that the computing cost is much smal…
We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution. This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates. By tracking the change …
VAEs improve representation learning by inverting the data-generating process through self-consistency.
problem VAEs struggle to invert the data-generating process, yet often succeed in representation learning.
method Studied VAEs in the limit of near-deterministic decoders, proving self-consistency and showing ELBO convergence to a regularized log-likelihood.
result VAEs can perform independent mechanism analysis (IMA), recovering true latent factors under specific conditions.
We consider the structured-output prediction problem through probabilistic approaches and generalize the "perturb-and-MAP" framework to more challenging weighted Hamming losses, which are crucial in applications. While in principle our approach is a straightforward marginalization, it requires solving many related MAP …
Bayesian scores improve structure learning in probabilistic circuits.
problem Improper structure learning in probabilistic circuits based on heuristics.
method Developed Bayesian structure scores for deterministic PCs, using them in a greedy cutset algorithm.
result Effective protection against overfitting and fast, almost hyper-parameter-free structure learner.
Bayesian evidence helps compare models but can overfit.
problem Comparing hypotheses consistent with observations.
method Marginal likelihood, Occam's razor, PAC-Bayes bounds.
result Marginal likelihood can negatively correlate with generalization.
Speeding up Markov Chain Monte Carlo (MCMC) for datasets with many observations by data subsampling has recently received considerable attention. A pseudo-marginal MCMC method is proposed that estimates the likelihood by data subsampling using a block-Poisson estimator. The estimator is a product of Poisson estimators,…
This study compares MC and QMC methods for likelihood functions.
problem Approximating the normalizing constant of posterior distributions and marginal likelihoods.
method Characterizes the integration error of MC and QMC methods for likelihood functions.
result QMC outperforms MC under certain conditions, especially in high dimensions.
Conservation laws improve diffusion model training by optimizing likelihood.
problem Training diffusion models with denoising objectives.
method Developed conservation laws based on GEXIT functions for memoryless noise processes.
result Unified characterization of diffusion model likelihood, reducing training to learning marginal posteriors.
New model generates graphs with tighter likelihood bounds and better quality.
problem Intractable likelihood of autoregressive graph models.
method Derive exact joint probability, approximate node orderings, variational inference.
result Lower bound on log-likelihood is significantly tighter than previous methods.
New method resolves nonidentifiability in mixture models.
problem Nonidentifiability in marginal models of mixture models.
method Introducing an effective temperature to generalize the marginal likelihood.
result Maximization of the generalized likelihood leads to unique results.
New method estimates marginal likelihood for deep learning models using training data alone.
problem Estimation difficulties in marginal likelihood for model selection in deep learning.
method Scalable marginal likelihood estimation based on Laplace's method and Gauss-Newton approximations.
result Estimate outperforms cross-validation and manual tuning on various datasets.
Evidential Softmax preserves multimodality in sparse probability distributions for generative models.
problem Sparse probability distributions in deep generative models make exact marginalization computationally intractable.
method Introduce ev-softmax, a sparse normalization function that preserves multimodality and can be trained with probabilistic loss functions.
result ev-softmax outperforms existing techniques in distributional accuracy and dimensionality reduction.
Corrects pseudo log-likelihood method issues in various applications.
problem Log-likelihood function unbounded issues in pseudo log-likelihood methods.
method Provided a counterexample and corrected algorithms in previous literature.
result Ensured well-definedness of maximum pseudo log-likelihood estimation.
A new method for efficient Gaussian process regression reduces complexity and improves scalability.
problem Efficient Gaussian process regression for large datasets.
method Learnable coreset-based variational inference for Gaussian processes.
result CVGP reduces the dimensionality of the variational parameter search space to linear complexity.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
Stochastic Gradient Annealed Importance Sampling for Efficient Online Marginal Likelihood Estimationstat.ML
We consider estimating the marginal likelihood in settings with independent and identically distributed (i.i.d.) data. We propose estimating the predictive distributions in a sequential factorization of the marginal likelihood in such settings by using stochastic gradient Markov Chain Monte Carlo techniques. This appro…
When used as a surrogate objective for maximum likelihood estimation in latent variable models, the evidence lower bound (ELBO) produces state-of-the-art results. Inspired by this, we consider the extension of the ELBO to a family of lower bounds defined by a particle filter's estimator of the marginal likelihood, the …
Bayesian approach learns invariances from data alone, but last layer approximation is not always sufficient.
problem Learning invariances in neural networks using only training data.
method Bayesian marginal likelihood for last layer, custom optimisation routine, new lower bound.
result Partial success on standard benchmarks and medical imaging dataset, failure on CIFAR10.
Due to the intractable partition function, the exact likelihood function for a Markov random field (MRF), in many situations, can only be approximated. Major approximation approaches include pseudolikelihood and Laplace approximation. In this paper, we propose a novel way of approximating the likelihood function throug…
We discuss the connection between information and copula theories by showing that a copula can be employed to decompose the information content of a multivariate distribution into marginal and dependence components, with the latter quantified by the mutual information. We define the information excess as a measure of d…
A Bayesian GED-Gamma stochastic volatility model for return data: a marginal likelihood approachq-fin.ST
Several studies explore inferences based on stochastic volatility (SV) models, taking into account the stylized facts of return data. The common problem is that the latent parameters of many volatility models are high-dimensional and analytically intractable, which means inferences require approximations using, for exa…
We give polynomial-time algorithms for the exact computation of lowest-energy (ground) states, worst margin violators, log partition functions, and marginal edge probabilities in certain binary undirected graphical models. Our approach provides an interesting alternative to the well-known graph cut paradigm in that it …
This paper introduces Kernel-based Information Criterion (KIC) for model selection in regression analysis. The novel kernel-based complexity measure in KIC efficiently computes the interdependency between parameters of the model using a variable-wise variance and yields selection of better, more robust regressors. Expe…
Factorized information criterion (FIC) is a recently developed approximation technique for the marginal log-likelihood, which provides an automatic model selection framework for a few latent variable models (LVMs) with tractable inference algorithms. This paper reconsiders FIC and fills theoretical gaps of previous FIC…
Learning latent variable models with stochastic variational inference is challenging when the approximate posterior is far from the true posterior, due to high variance in the gradient estimates. We propose a novel rejection sampling step that discards samples from the variational posterior which are assigned low likel…
Log-linear models are the popular workhorses of analyzing contingency tables. A log-linear parameterization of an interaction model can be more expressive than a direct parameterization based on probabilities, leading to a powerful way of defining restrictions derived from marginal, conditional and context-specific ind…
We introduce the multiresolution recurrent neural network, which extends the sequence-to-sequence framework to model natural language generation as two parallel discrete stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language tokens. There are many ways to estimate or learn the …