Maximum likelihood training improves the performance of score-based diffusion models.
problem Training score-based diffusion models with maximum likelihood.
method Trained by minimizing a weighted combination of score matching losses, with a specific weighting scheme that bounds negative log-likelihood.
result Maximum likelihood training improves the log-likelihood of score-based diffusion models across multiple datasets.
Expectation Maximization (EM) is among the most popular algorithms for maximum likelihood estimation, but it is generally only guaranteed to find its stationary points of the log-likelihood objective. The goal of this article is to present theoretical and empirical evidence that over-parameterization can help EM avoid …
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
RWR converges to global optimum in certain settings.
problem Proving convergence of RWR to optimal policy.
method Iterative learning with return-weighted log-likelihood.
result RWR converges to global optimum under certain conditions.
Improved convergence rates for MLE in mixture models using penalized log-likelihood.
problem Convergence rates for MLE in finite mixture models.
method Penalizing log-likelihood to discourage vanishing mixing weights, using Wasserstein distance and new loss functions.
result Improved convergence rates for some mixture components, faster than traditional methods.
The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distributi…
A determinantal point process (DPP) is a probabilistic model of set diversity compactly parameterized by a positive semi-definite kernel matrix. To fit a DPP to a given task, we would like to learn the entries of its kernel matrix by maximizing the log-likelihood of the available data. However, log-likelihood is non-co…
The log-likelihood loss in heteroscedastic neural networks can lead to poor parameter estimates.
problem Capturing aleatoric uncertainty in deep learning models.
method Examine the log-likelihood loss in conjunction with gradient-based optimizers and propose an alternative formulation, β-NLL. result Using an appropriate β largely mitigates the issue of poor parameter estimates. Innovative game theory approach optimizes survival analysis metrics.
problem Survival analysis models trained with maximum likelihood do not directly optimize criteria like Brier score or Bernoulli log likelihood.
method Inverse-Weighted Survival Games: Construct objectives from re-weighted estimates featuring the other model, holding the latter fixed during training.
result Games optimize Brier score on simulations and real-world data.
This paper improves SNN training by using multiple sample compartments.
problem Training SNNs with single-sample estimators leads to inaccurate log-likelihood estimates.
method Proposes a GEM-based online learning algorithm that uses multiple independent spiking signals.
result Significant improvements in log-likelihood, accuracy, and calibration with multiple compartments.
wd1 improves reasoning in dLLMs by optimizing policies without policy ratios.
problem Improving reasoning in diffusion-based large language models through RL.
method wd1: ratio-free policy optimization using weighted log-likelihood.
result wd1 outperforms diffusion-based GRPO while requiring lower computational cost.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
Corrects pseudo log-likelihood method issues in various applications.
problem Log-likelihood function unbounded issues in pseudo log-likelihood methods.
method Provided a counterexample and corrected algorithms in previous literature.
result Ensured well-definedness of maximum pseudo log-likelihood estimation.
New Holder bounds improve variational inference by flattening thermodynamic curves.
problem Improving variational inference by addressing performance gaps between theory and practice.
method Generalizing thermodynamic integration to weighted Holder mean, introducing Holder bounds.
result Holder bounds promise a one-step approximation of exact marginal log-likelihood.
Coherent uncertainty quantification is a key strength of Bayesian methods. But modern algorithms for approximate Bayesian posterior inference often sacrifice accurate posterior uncertainty estimation in the pursuit of scalability. This work shows that previous Bayesian coreset construction algorithms---which build a sm…
Introduces a new theoretical framework for exponential smoothing.
problem Theoretical foundation and robustness of simple exponential smoothing.
method Stochastic gradient ascent to optimize Gaussian log-likelihood functions.
result Simple exponential smoothing converges to the trend of a trend-stationary process.
A new VIS approach improves log-likelihood estimation in latent variable models.
problem Challenges in achieving high log-likelihood with VI for complex posterior distributions.
method Uses forward χ2 divergence to optimize proposal distribution for better log-likelihood estimation. result Consistently outperforms state-of-the-art baselines in log-likelihood and parameter estimation.
New method optimizes clustering with better log-likelihood landscape.
problem Nonconvex log-likelihood optimization in model-based clustering.
method Entropic optimal transport loss for Sinkhorn-EM algorithm.
result New loss function avoids spurious local optima.
New method calculates DMN log-likelihood faster.
problem Precise and fast computation of DMN log-likelihood.
method Derived a closed form expression using gamma function properties.
result Closed form calculation is faster with same accuracy.
This work improves bounds on Bayesian coreset quality.
problem Limitations of existing theoretical analysis of Bayesian coresets.
method Develops general upper and lower bounds on KL divergence.
result Demonstrates flexibility of new theoretical bounds in various models.
Variational Laplace improves Bayesian neural networks performance.
problem Improving Bayesian neural networks performance.
method Develops variational Laplace for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms other inference methods.
A new KDE model prevents singular solutions and accelerates optimization for probabilistic modeling.
problem Adapting to varying densities in data regions for probabilistic modeling.
method Adaptive KDE model with individual bandwidths, LOO-MLL criterion, and modified EM algorithm.
result The proposed models prevent singular solutions and have promising performance.
The Restricted Boltzmann Machines (RBM) can be used either as classifiers or as generative models. The quality of the generative RBM is measured through the average log-likelihood on test data. Due to the high computational complexity of evaluating the partition function, exact calculation of test log-likelihood is ver…
Variational Laplace improves Bayesian neural network performance without sampling.
problem Improving Bayesian neural network performance and calibration.
method Develops a new variational Laplace method for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms standard VI methods in test performance and calibration.
We consider the structured-output prediction problem through probabilistic approaches and generalize the "perturb-and-MAP" framework to more challenging weighted Hamming losses, which are crucial in applications. While in principle our approach is a straightforward marginalization, it requires solving many related MAP …
Method completes mixed matrix from complex surveys with heterogeneous missingness.
problem Recovering a mixed dataframe matrix from complex survey sampling with different missingness patterns.
method Two-stage procedure: logistic regression for missingness modeling, and weighted log-likelihood maximization with low-rank constraint.
result The proposed method achieves sublinear convergence and shows superior performance compared to existing methods.
Multi-task learning is a method for improving the generalizability of multiple tasks. In order to perform multiple classification tasks with one neural network model, the losses of each task should be combined. Previous studies have mostly focused on multiple prediction tasks using joint loss with static weights for tr…
The fate of scientific hypotheses often relies on the ability of a computational model to explain the data, quantified in modern statistical approaches by the likelihood function. The log-likelihood is the key element for parameter estimation and model evaluation. However, the log-likelihood of complex models in fields…
This paper introduces a method to estimate log-likelihood in VAE models.
problem Difficulty in comparing models trained via ELBO due to lack of log-likelihood.
method Introduces a general upper bound to approximate model evidence.
result Efficiently approximates model evidence and compares to other bounds.
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…
We explain SSL objectives as log-likelihoods in a data curation model.
problem Lack of understanding of SSL objectives as log-likelihoods.
method Formulate SSL objectives as a log-likelihood in a generative model of data curation.
result SSL methods can be understood as lower-bounds on a principled log-likelihood.
Deep model predicts wildfire data in incomplete regions.
problem Predicting distributions of wildfire frequency and size in missing data regions.
method Conditional Missing data Importance-Weighted Autoencoder (CMIWAE) model trained on incomplete data.
result Maximizes log-likelihood of observed wildfire information.
This work evaluates deep generative models using RD curves, providing a more comprehensive quality assessment.
problem Quantitative evaluation of deep generative models is challenging, especially for implicit models.
method Proposes using rate distortion (RD) curves to evaluate and compare deep generative models, approximating the entire curve with similar computations to log-likelihood estimation.
result Approximating the entire RD curve provides a more comprehensive quality assessment than scalar-valued metrics.
Enhances VAEs for sharper image synthesis.
problem Blurriness in generated images from VAEs.
method Integrates a downscaled version of the original image into the VAE framework and uses it as input to the decoder.
result Improves FID score in image synthesis while maintaining similar log-likelihood performance.
A new method normalizes EBM training by introducing a learnable parameter.
problem Training energy-based models with maximum likelihood is challenging due to intractable normalisation constants.
method Proposes a self-normalised log-likelihood (SNL) objective that introduces a learnable parameter representing the normalisation constant.
result The SNL objective is a lower bound of the log-likelihood and can be directly optimised using stochastic gradient techniques.
In this paper we revisit the weighted likelihood bootstrap, a method that generates samples from an approximate Bayesian posterior of a parametric model. We show that the same method can be derived, without approximation, under a Bayesian nonparametric model with the parameter of interest defined as minimising an expec…
This work gives an in-depth derivation of the trainable evidence lower bound obtained from the marginal joint log-Likelihood with the goal of training a Multi-Modal Variational Autoencoder (M2VAE).
A common strategy for sparse linear regression is to introduce regularization, which eliminates irrelevant features by letting the corresponding weights be zeros. However, regularization often shrinks the estimator for relevant features, which leads to incorrect feature selection. Motivated by the above-mentioned issue…
We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly …
EM algorithm converges to global max in latent Gaussian tree models.
problem Optimizing log-likelihood in latent Gaussian tree models.
method Analyzed the optimization landscape and convergence of EM algorithm.
result EM algorithm converges to global maximum in latent Gaussian tree models.
Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
problem Estimating log-likelihood of complex multivariate point processes is computationally expensive.
method Noise-Contrastive Estimation adapted for multivariate point processes, with provable guarantees.
result Our method achieves similar log-likelihood with fewer evaluations and less time.
In this work, a deep learning-based method for log-likelihood ratio (LLR) lossy compression and quantization is proposed, with emphasis on a single-input single-output uncorrelated fading communication setting. A deep autoencoder network is trained to compress, quantize and reconstruct the bit log-likelihood ratios cor…
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 con…
The MAP estimate's log-likelihood sub-optimality is hard to bound in general.
problem Bounding the expected log-likelihood sub-optimality of MAP for exponential families.
method Interpreting MAP as stochastic mirror descent and analyzing convergence rates.
result Current convergence results do not apply to standard examples of exponential families.
Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses and neural activity together is still lacking. The computational …
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equival…
Proposes a non-convex optimization method for a parsimonious weighted naive Bayes classifier.
problem Improving naïve Bayes classifier performance with a large number of input variables.
method Sparse regularization of model log-likelihood for direct estimation of variable weights.
result Optimization-based weighted naïve Bayes classifiers achieve equivalent performance to averaging-based classifiers.
A study on optimizing data augmentation weights for improved test-time predictions.
problem Improving robustness of predictions during testing with data augmentation methods.
method A weighted Test-Time Augmentation (TTA) approach based on variational Bayesian framework to optimize weights.
result Optimizing weights suppresses unwanted data augmentations and improves prediction performance.