Corrects pseudo log-likelihood method issues in various applications.
problem Log-likelihood function unbounded issues in pseudo log-likelihood methods.
method Provided a counterexample and corrected algorithms in previous literature.
result Ensured well-definedness of maximum pseudo log-likelihood estimation.
IBS efficiently estimates log-likelihood without bias.
problem Estimating log-likelihood for complex models.
method Inverse Binomial Sampling (IBS) for efficient, unbiased estimation.
result IBS produces lower error in estimated parameters and maximum log-likelihood values.
A new VIS approach improves log-likelihood estimation in latent variable models.
problem Challenges in achieving high log-likelihood with VI for complex posterior distributions.
method Uses forward χ2 divergence to optimize proposal distribution for better log-likelihood estimation. result Consistently outperforms state-of-the-art baselines in log-likelihood and parameter estimation.
The Restricted Boltzmann Machines (RBM) can be used either as classifiers or as generative models. The quality of the generative RBM is measured through the average log-likelihood on test data. Due to the high computational complexity of evaluating the partition function, exact calculation of test log-likelihood is ver…
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…
Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
problem Estimating log-likelihood of complex multivariate point processes is computationally expensive.
method Noise-Contrastive Estimation adapted for multivariate point processes, with provable guarantees.
result Our method achieves similar log-likelihood with fewer evaluations and less time.
This work evaluates deep generative models using RD curves, providing a more comprehensive quality assessment.
problem Quantitative evaluation of deep generative models is challenging, especially for implicit models.
method Proposes using rate distortion (RD) curves to evaluate and compare deep generative models, approximating the entire curve with similar computations to log-likelihood estimation.
result Approximating the entire RD curve provides a more comprehensive quality assessment than scalar-valued metrics.
Estimates log-likelihood of interacting particle systems using virtual particles.
problem Inconsistent estimation of finite-particle log-likelihood in large particle systems.
method Stochastic gradient estimate using continuous trajectory and virtual particle systems.
result Convergence to stationary points of limiting mean-field system's log-likelihood.
The log-likelihood loss in heteroscedastic neural networks can lead to poor parameter estimates.
problem Capturing aleatoric uncertainty in deep learning models.
method Examine the log-likelihood loss in conjunction with gradient-based optimizers and propose an alternative formulation, β-NLL. result Using an appropriate β largely mitigates the issue of poor parameter estimates. This paper introduces a method to estimate log-likelihood in VAE models.
problem Difficulty in comparing models trained via ELBO due to lack of log-likelihood.
method Introduces a general upper bound to approximate model evidence.
result Efficiently approximates model evidence and compares to other bounds.
A new method normalizes EBM training by introducing a learnable parameter.
problem Training energy-based models with maximum likelihood is challenging due to intractable normalisation constants.
method Proposes a self-normalised log-likelihood (SNL) objective that introduces a learnable parameter representing the normalisation constant.
result The SNL objective is a lower bound of the log-likelihood and can be directly optimised using stochastic gradient techniques.
Paper develops unbiased gradient estimator for continuous-time models.
problem Estimating unbiased gradient of log-likelihood for continuous-time models.
method Doubly randomized scheme with coupled conditional particle filter (CCPF).
result Unbiased gradient estimate facilitates gradient-based algorithms.
The MAP estimate's log-likelihood sub-optimality is hard to bound in general.
problem Bounding the expected log-likelihood sub-optimality of MAP for exponential families.
method Interpreting MAP as stochastic mirror descent and analyzing convergence rates.
result Current convergence results do not apply to standard examples of exponential families.
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
This paper addresses the estimation of parameters of a Bayesian network from incomplete data. The task is usually tackled by running the Expectation-Maximization (EM) algorithm several times in order to obtain a high log-likelihood estimate. We argue that choosing the maximum log-likelihood estimate (as well as the max…
A method to approximate posterior distributions using Monte Carlo and variational inference.
problem Lack of systematic understanding of how optimizing different objectives relates to approximating the posterior distribution.
method Divide and couple procedure to identify augmented proposal and target distributions.
result Maximizing the VI objective leads to an augmented variational distribution that approximates the posterior distribution.
Aligns databases with Gaussian features using MAP estimation and thresholding.
problem Aligning databases with jointly Gaussian features.
method Two algorithms: MAP estimation and thresholding of log likelihood ratios.
result Conditions on mutual information for reliable performance of algorithms.
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
This paper improves SNN training by using multiple sample compartments.
problem Training SNNs with single-sample estimators leads to inaccurate log-likelihood estimates.
method Proposes a GEM-based online learning algorithm that uses multiple independent spiking signals.
result Significant improvements in log-likelihood, accuracy, and calibration with multiple compartments.
A new algorithm identifies outliers in Gaussian clustering models.
problem Handling outliers in Gaussian model-based clustering.
method OCLUST algorithm removes least plausible points based on subset log-likelihoods until they adhere to a reference distribution.
result OCLUST inherently estimates the number of outliers.
New method for online learning in interacting particle systems.
problem Parameter estimation in stochastic interacting particle systems.
method Stochastic approximation of gradient of asymptotic log likelihood using continuous observations.
result Convergence to stationary points of asymptotic log-likelihood under suitable assumptions.
Maximum likelihood (ML) estimation using Newton's method in nonlinear state space models (SSMs) is a challenging problem due to the analytical intractability of the log-likelihood and its gradient and Hessian. We estimate the gradient and Hessian using Fisher's identity in combination with a smoothing algorithm. We exp…
Recently, there has been focus on penalized log-likelihood covariance estimation for sparse inverse covariance (precision) matrices. The penalty is responsible for inducing sparsity, and a very common choice is the convex l1 norm. However, the best estimator performance is not always achieved with this penalty. The …
PresGANs improve GANs by mitigating mode collapse and enhancing log-likelihood.
problem GANs struggle with mode collapse and lack a reliable way to evaluate generalization.
method PresGANs add noise to density networks and use entropy regularization to stabilize training and capture all modes.
result PresGANs reduce the gap in predictive log-likelihood between GANs and VAEs.
Method infers parameters in complex diffusion processes.
problem Parameter inference in high-dimensional, non-linear diffusion processes.
method Differentiable score matching to approximate diffusion bridges, used in an importance sampler.
result Numerically stable framework for parameter inference and diffusion mean estimation.
Paper proposes efficient and accurate initialization and EM algorithm for PL mixture models.
problem Initialization issues and combinatorial complexity in PL likelihood maximization.
method Initialization algorithm and EM algorithm for true log-likelihood maximization.
result Proposed algorithm provides accurate initial estimates and efficiently maximizes true log-likelihood.
EQ-Net combines LLR estimation and quantization using deep learning.
problem Unified solution for LLR estimation and quantization.
method Two-stage algorithm using LLR compression as a pretext task.
result Achieves state-of-the-art results with gains in efficiency and latency.
New lower bounds improve logistic log-likelihood optimization and inference.
problem Designing computationally tractable lower bounds for logistic log-likelihoods.
method Developed a piece-wise quadratic lower bound that uniformly improves tangent quadratic minorizers.
result Improves the speed of convergence and accuracy of variational Bayes approximations.
Parallel Gaussian process surrogate for noisy likelihood evaluations in Bayesian inference.
problem Bayesian inference with limited noisy log-likelihood evaluations from complex models.
method Hierarchical Gaussian process surrogate model for log-likelihood, batch-sequential design strategies.
result Robust, highly parallelizable, and sample-efficient method.
A novel rejection sampling step improves variational inference for latent variable models.
problem High variance in gradient estimates for approximate posterior in stochastic variational inference.
method Rejection sampling to discard low-likelihood samples and a new gradient estimator.
result Improves marginal log-likelihood estimation by 3.71 nats and 0.21 nats.
Adaptive estimation for nonstationary time series reduces computational cost.
problem Estimating parameters of nonstationary time series with varying parameters over time.
method Moving exponential moving ML estimator for scale parameter estimation.
result Significantly improved log-likelihoods compared to standard estimation.
Partition Tree estimates conditional densities for mixed continuous and categorical variables.
problem Estimating conditional densities for mixed data types.
method Tree-based framework modeling conditional distributions as piecewise-constant densities on adaptive partitions, minimizing conditional negative log-likelihood.
result Improved probabilistic prediction compared to CART-style trees and state-of-the-art methods.
We propose a novel method for maximum likelihood-based parameter inference in nonlinear and/or non-Gaussian state space models. The method is an iterative procedure with three steps. At each iteration a particle filter is used to estimate the value of the log-likelihood function at the current parameter iterate. Using …
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
Deep learning predicts kidney graft survival better than traditional methods.
problem Improving survival analysis for kidney graft patients.
method Deep learning method that jointly predicts survival times and ranks.
result Deep learning outperforms Cox Proportional Hazards and other methods.
We propose Subsampling MCMC, a Markov Chain Monte Carlo (MCMC) framework where the likelihood function for n observations is estimated from a random subset of m observations. We introduce a highly efficient unbiased estimator of the log-likelihood based on control variates, such that the computing cost is much smal…
RED models efficiently estimate distributions of real-valued data.
problem Efficient modeling of real-valued data with conditional dependencies.
method Semiparametric modeling using recurrent neural networks (RNNs).
result RED models achieve lower held-out negative log-likelihood than other neural network approaches.
Proposes a new multi-scale architecture for generative flows to improve log-likelihood and sampling quality.
problem Challenges of high-dimensional latent space in flow models.
method Data-dependent dimension factorization based on likelihood contribution heuristic.
result Improvements in log-likelihood score and sampling quality on image benchmarks.
Improved convergence rates for MLE in mixture models using penalized log-likelihood.
problem Convergence rates for MLE in finite mixture models.
method Penalizing log-likelihood to discourage vanishing mixing weights, using Wasserstein distance and new loss functions.
result Improved convergence rates for some mixture components, faster than traditional methods.
We simplify matrix computations for block matrices, especially useful for covariance and correlation matrices.
problem Complex computations for block matrices, especially for covariance and correlation matrices.
method Obtained a canonical representation for block matrices, facilitating computation of various matrix operations.
result Simplified computation of matrix operations for block matrices, particularly useful for covariance and correlation matrices.
Derives M2VAE objective from marginal joint log-likelihood.
problem Training Multi-Modal Variational Autoencoders (M2VAEs). method Derives trainable evidence lower bound from marginal joint log-likelihood.
result Derives M2VAE objective from marginal joint log-likelihood. SPRT-TANDEM improves sequential classification accuracy with fewer samples.
problem Efficiently classifying sequential data with high accuracy and low sampling cost.
method Deep neural network-based SPRT algorithm that estimates log-likelihood ratio of two hypotheses.
result SPRT-TANDEM achieves statistically significantly better classification accuracy than other classifiers with fewer samples.
New method optimizes clustering with better log-likelihood landscape.
problem Nonconvex log-likelihood optimization in model-based clustering.
method Entropic optimal transport loss for Sinkhorn-EM algorithm.
result New loss function avoids spurious local optima.
New method estimates partition functions for complex distributions.
problem Computing partition functions for complex and multimodal distributions.
method Rao-Blackwellized Tempered Sampling
result Empirically more accurate than Annealed Importance Sampling.
New method calculates DMN log-likelihood faster.
problem Precise and fast computation of DMN log-likelihood.
method Derived a closed form expression using gamma function properties.
result Closed form calculation is faster with same accuracy.
DNLL loss improves deep LDA accuracy and consistency.
problem Pathological solutions in unconstrained Deep LDA.
method Introducing Discriminative Negative Log-Likelihood (DNLL) loss.
result Deep LDA trained with DNLL produces clean latent spaces and better calibrated probabilities.
Improvement guarantees for semi-supervised classifiers can currently only be given under restrictive conditions on the data. We propose a general way to perform semi-supervised parameter estimation for likelihood-based classifiers for which, on the full training set, the estimates are never worse than the supervised so…
GBHT uses gradient boosting for density estimation with theoretical guarantees.
problem Density estimation for unsupervised learning.
method Gradient Boosting Histogram Transform (GBHT) with Negative Log Likelihood loss.
result GBHT achieves faster convergence rates and better performance than base learners in density estimation.