Corrects pseudo log-likelihood method issues in various applications.
problem Log-likelihood function unbounded issues in pseudo log-likelihood methods.
method Provided a counterexample and corrected algorithms in previous literature.
result Ensured well-definedness of maximum pseudo log-likelihood estimation.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
Estimates log marginal likelihood using multilevel Monte Carlo.
problem Estimating log marginal likelihood accurately.
method Unbiased multilevel Monte Carlo estimator.
result Validates application in variational Bayes.
A new VIS approach improves log-likelihood estimation in latent variable models.
problem Challenges in achieving high log-likelihood with VI for complex posterior distributions.
method Uses forward χ2 divergence to optimize proposal distribution for better log-likelihood estimation. result Consistently outperforms state-of-the-art baselines in log-likelihood and parameter estimation.
IBS efficiently estimates log-likelihood without bias.
problem Estimating log-likelihood for complex models.
method Inverse Binomial Sampling (IBS) for efficient, unbiased estimation.
result IBS produces lower error in estimated parameters and maximum log-likelihood values.
New method optimizes clustering with better log-likelihood landscape.
problem Nonconvex log-likelihood optimization in model-based clustering.
method Entropic optimal transport loss for Sinkhorn-EM algorithm.
result New loss function avoids spurious local optima.
New method calculates DMN log-likelihood faster.
problem Precise and fast computation of DMN log-likelihood.
method Derived a closed form expression using gamma function properties.
result Closed form calculation is faster with same accuracy.
Log-concavity proven for multinomial likelihoods under specific constraints.
problem Log-concavity of multinomial likelihoods under interval censoring constraints.
method Proved log-concavity by showing M-convex subsets of the discrete simplex.
result Likelihood function is completely log-concave.
Maximum likelihood training improves the performance of score-based diffusion models.
problem Training score-based diffusion models with maximum likelihood.
method Trained by minimizing a weighted combination of score matching losses, with a specific weighting scheme that bounds negative log-likelihood.
result Maximum likelihood training improves the log-likelihood of score-based diffusion models across multiple datasets.
The Restricted Boltzmann Machines (RBM) can be used either as classifiers or as generative models. The quality of the generative RBM is measured through the average log-likelihood on test data. Due to the high computational complexity of evaluating the partition function, exact calculation of test log-likelihood is ver…
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…
A new method normalizes EBM training by introducing a learnable parameter.
problem Training energy-based models with maximum likelihood is challenging due to intractable normalisation constants.
method Proposes a self-normalised log-likelihood (SNL) objective that introduces a learnable parameter representing the normalisation constant.
result The SNL objective is a lower bound of the log-likelihood and can be directly optimised using stochastic gradient techniques.
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
Improved Gaussian process regression with tighter log marginal likelihood bounds.
problem Improving predictive performance in Gaussian process regression models.
method Lower bound on log marginal likelihood using conjugate gradients.
result Improved predictive performance compared to other conjugate gradient based approaches.
We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly …
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equival…
This paper introduces a method to estimate log-likelihood in VAE models.
problem Difficulty in comparing models trained via ELBO due to lack of log-likelihood.
method Introduces a general upper bound to approximate model evidence.
result Efficiently approximates model evidence and compares to other bounds.
We explain SSL objectives as log-likelihoods in a data curation model.
problem Lack of understanding of SSL objectives as log-likelihoods.
method Formulate SSL objectives as a log-likelihood in a generative model of data curation.
result SSL methods can be understood as lower-bounds on a principled log-likelihood.
SUMO provides unbiased log marginal likelihood estimation for latent variable models.
problem Biased estimates of log marginal likelihood in latent variable models.
method Randomized truncation of infinite series for unbiased estimation.
result Models trained with SUMO give better test-set likelihoods than standard methods.
Paper shows how sparse inversion speeds up log determinant derivatives.
problem Deriving log determinant derivatives for sparse matrices.
method Sparse inversion, selected inversion, accelerates computation.
result Derivative of log determinant can be computed faster with sparse inversion.
Estimates log-likelihood of interacting particle systems using virtual particles.
problem Inconsistent estimation of finite-particle log-likelihood in large particle systems.
method Stochastic gradient estimate using continuous trajectory and virtual particle systems.
result Convergence to stationary points of limiting mean-field system's log-likelihood.
Study detects signals in spiked Wigner models using log likelihood ratio.
problem Detecting signals in rank-one spiked Wigner models with non-Gaussian noise.
method Proved asymptotic normality of log likelihood ratio and computed error thresholds.
result Optimal signal-to-noise ratio threshold for reliable detection.
Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
problem Estimating log-likelihood of complex multivariate point processes is computationally expensive.
method Noise-Contrastive Estimation adapted for multivariate point processes, with provable guarantees.
result Our method achieves similar log-likelihood with fewer evaluations and less time.
EM algorithm converges to global max in latent Gaussian tree models.
problem Optimizing log-likelihood in latent Gaussian tree models.
method Analyzed the optimization landscape and convergence of EM algorithm.
result EM algorithm converges to global maximum in latent Gaussian tree models.
Improves Gaussian process regression without bias.
problem Bias in Gaussian process regression estimates.
method Adaptive computation selection to minimize bias.
result Guaranteed small bias in log marginal likelihood estimates.
The MAP estimate's log-likelihood sub-optimality is hard to bound in general.
problem Bounding the expected log-likelihood sub-optimality of MAP for exponential families.
method Interpreting MAP as stochastic mirror descent and analyzing convergence rates.
result Current convergence results do not apply to standard examples of exponential families.
This work evaluates deep generative models using RD curves, providing a more comprehensive quality assessment.
problem Quantitative evaluation of deep generative models is challenging, especially for implicit models.
method Proposes using rate distortion (RD) curves to evaluate and compare deep generative models, approximating the entire curve with similar computations to log-likelihood estimation.
result Approximating the entire RD curve provides a more comprehensive quality assessment than scalar-valued metrics.
New method for robust distribution alignment using log-likelihood ratio and normalizing flows.
problem Distribution alignment challenges in deep learning.
method Log-likelihood ratio statistic and normalizing flows.
result Minimizing the proposed objective yields robust domain alignment.
The log-likelihood loss in heteroscedastic neural networks can lead to poor parameter estimates.
problem Capturing aleatoric uncertainty in deep learning models.
method Examine the log-likelihood loss in conjunction with gradient-based optimizers and propose an alternative formulation, β-NLL. result Using an appropriate β largely mitigates the issue of poor parameter estimates. Enhances VAEs for sharper image synthesis.
problem Blurriness in generated images from VAEs.
method Integrates a downscaled version of the original image into the VAE framework and uses it as input to the decoder.
result Improves FID score in image synthesis while maintaining similar log-likelihood performance.
This work gives an in-depth derivation of the trainable evidence lower bound obtained from the marginal joint log-Likelihood with the goal of training a Multi-Modal Variational Autoencoder (M2VAE).
Sparse matrices simplify computation of GP variances and likelihoods.
problem Efficient computation of posterior variance and log-likelihood for additive Matérn GPs.
method Represented posterior mean, variance, log-likelihood, and gradient using sparse matrices.
result Efficient computation of posterior mean, variance, log-likelihood, and gradient in O(nlogn) time. PresGANs improve GANs by mitigating mode collapse and enhancing log-likelihood.
problem GANs struggle with mode collapse and lack a reliable way to evaluate generalization.
method PresGANs add noise to density networks and use entropy regularization to stabilize training and capture all modes.
result PresGANs reduce the gap in predictive log-likelihood between GANs and VAEs.
In this work, a deep learning-based method for log-likelihood ratio (LLR) lossy compression and quantization is proposed, with emphasis on a single-input single-output uncorrelated fading communication setting. A deep autoencoder network is trained to compress, quantize and reconstruct the bit log-likelihood ratios cor…
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 con…
Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poore…
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from co…
Proves error bounds for PGD, extending log-Sobolev and Talagrand inequalities.
problem Maximum likelihood estimation of large latent variable models.
method Extending log-Sobolev and Talagrand inequalities to models with strongly concave log-likelihoods.
result Non-asymptotic error bounds for PGD in models satisfying LSI and PŁI.
Framework for Bayesian inference using GP emulated MH sampler for noisy likelihoods.
problem Approximate Bayesian inference with limited noisy log-likelihood evaluations.
method Gaussian process emulates MH sampler for log-likelihood evaluations; sequential experimental design selects evaluation points.
result Approximate sampler is sample-efficient and robust to GP assumptions.
Recent work in variational inference (VI) uses ideas from Monte Carlo estimation to tighten the lower bounds on the log-likelihood that are used as objectives. However, there is no systematic understanding of how optimizing different objectives relates to approximating the posterior distribution. Developing such a conn…
This paper improves SNN training by using multiple sample compartments.
problem Training SNNs with single-sample estimators leads to inaccurate log-likelihood estimates.
method Proposes a GEM-based online learning algorithm that uses multiple independent spiking signals.
result Significant improvements in log-likelihood, accuracy, and calibration with multiple compartments.
We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanatio…
Improved DDPMs achieve high log-likelihoods and sample quality with fewer passes.
problem Improving log-likelihoods of DDPMs while maintaining high sample quality.
method Modifying DDPMs with learning variances and using precision-recall metrics.
result DDPMs can achieve competitive log-likelihoods with fewer forward passes.
New lower bounds improve logistic log-likelihood optimization and inference.
problem Designing computationally tractable lower bounds for logistic log-likelihoods.
method Developed a piece-wise quadratic lower bound that uniformly improves tangent quadratic minorizers.
result Improves the speed of convergence and accuracy of variational Bayes approximations.
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of th…
Evaluation of the marginal likelihood plays an important role in model selection problems. The widely applicable Bayesian information criterion (WBIC) and singular Bayesian information criterion (sBIC) give approximations to the log marginal likelihood, which can be applied to both regular and singular models. When the…
Clustering, or unsupervised classification, is a task often plagued by outliers. Yet there is a paucity of work on handling outliers in clustering. Outlier identification algorithms tend to fall into three broad categories: outlier inclusion, outlier trimming, and post hoc outlier identification methods, with the forme…