Adaptive t-distribution estimates nonstationary time series using moving moments.
problem Nonstationary time series with varying dependence structure.
method Moving estimator optimizing a weighted log-likelihood, using exponential moving averages for moments.
result Evolution of ν parameter in Student's t-distribution, capturing tail behavior and extreme events.
Adaptive estimation for nonstationary time series reduces computational cost.
problem Estimating parameters of nonstationary time series with varying parameters over time.
method Moving exponential moving ML estimator for scale parameter estimation.
result Significantly improved log-likelihoods compared to standard estimation.
A new MFG framework for evolving clusters from Gaussian mixtures.
problem Evolutionary clustering of time-dependent Gaussian mixtures.
method Control-theoretic framework based on Mean Field Games (MFG) with coupled HJB and Fokker-Planck systems.
result MFG dynamics recover classical EM algorithm trajectories with mass conservation.
Adaptive estimation of alpha-Stable distribution and Hurst exponent for nonstationary time series.
problem Nonstationary time series require adaptive models to avoid bias.
method Moving estimator with exponentially weakening weights of old values, optimized using EMA of absolute central moments.
result Continuous adaptive estimation of alpha-Stable distribution and Hurst exponent for market stability evaluation.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
Corrects pseudo log-likelihood method issues in various applications.
problem Log-likelihood function unbounded issues in pseudo log-likelihood methods.
method Provided a counterexample and corrected algorithms in previous literature.
result Ensured well-definedness of maximum pseudo log-likelihood estimation.
A new VIS approach improves log-likelihood estimation in latent variable models.
problem Challenges in achieving high log-likelihood with VI for complex posterior distributions.
method Uses forward χ2 divergence to optimize proposal distribution for better log-likelihood estimation. result Consistently outperforms state-of-the-art baselines in log-likelihood and parameter estimation.
New method optimizes clustering with better log-likelihood landscape.
problem Nonconvex log-likelihood optimization in model-based clustering.
method Entropic optimal transport loss for Sinkhorn-EM algorithm.
result New loss function avoids spurious local optima.
New method calculates DMN log-likelihood faster.
problem Precise and fast computation of DMN log-likelihood.
method Derived a closed form expression using gamma function properties.
result Closed form calculation is faster with same accuracy.
The Restricted Boltzmann Machines (RBM) can be used either as classifiers or as generative models. The quality of the generative RBM is measured through the average log-likelihood on test data. Due to the high computational complexity of evaluating the partition function, exact calculation of test log-likelihood is ver…
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
We consider the problem of recovering an expert's reward function with inverse reinforcement learning (IRL) when there are missing/incomplete state-action pairs or observations in the demonstrated trajectories. This issue of missing trajectory data or information occurs in many situations, e.g., GPS signals from vehicl…
The fate of scientific hypotheses often relies on the ability of a computational model to explain the data, quantified in modern statistical approaches by the likelihood function. The log-likelihood is the key element for parameter estimation and model evaluation. However, the log-likelihood of complex models in fields…
This paper introduces a method to estimate log-likelihood in VAE models.
problem Difficulty in comparing models trained via ELBO due to lack of log-likelihood.
method Introduces a general upper bound to approximate model evidence.
result Efficiently approximates model evidence and compares to other bounds.
A new method normalizes flow mixtures for better inference across different data types.
problem Inference failure across diverse posterior geometries in normalizing flows.
method Introduces a two-stage framework with a stable global weighting mechanism based on sEMA.
result Achieves consistent NLL improvements and stable weight trajectories over baselines.
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…
We explain SSL objectives as log-likelihoods in a data curation model.
problem Lack of understanding of SSL objectives as log-likelihoods.
method Formulate SSL objectives as a log-likelihood in a generative model of data curation.
result SSL methods can be understood as lower-bounds on a principled log-likelihood.
Maximum likelihood training improves the performance of score-based diffusion models.
problem Training score-based diffusion models with maximum likelihood.
method Trained by minimizing a weighted combination of score matching losses, with a specific weighting scheme that bounds negative log-likelihood.
result Maximum likelihood training improves the log-likelihood of score-based diffusion models across multiple datasets.
This work evaluates deep generative models using RD curves, providing a more comprehensive quality assessment.
problem Quantitative evaluation of deep generative models is challenging, especially for implicit models.
method Proposes using rate distortion (RD) curves to evaluate and compare deep generative models, approximating the entire curve with similar computations to log-likelihood estimation.
result Approximating the entire RD curve provides a more comprehensive quality assessment than scalar-valued metrics.
Enhances VAEs for sharper image synthesis.
problem Blurriness in generated images from VAEs.
method Integrates a downscaled version of the original image into the VAE framework and uses it as input to the decoder.
result Improves FID score in image synthesis while maintaining similar log-likelihood performance.
A new method normalizes EBM training by introducing a learnable parameter.
problem Training energy-based models with maximum likelihood is challenging due to intractable normalisation constants.
method Proposes a self-normalised log-likelihood (SNL) objective that introduces a learnable parameter representing the normalisation constant.
result The SNL objective is a lower bound of the log-likelihood and can be directly optimised using stochastic gradient techniques.
This work gives an in-depth derivation of the trainable evidence lower bound obtained from the marginal joint log-Likelihood with the goal of training a Multi-Modal Variational Autoencoder (M2VAE).
We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly …
EM algorithm converges to global max in latent Gaussian tree models.
problem Optimizing log-likelihood in latent Gaussian tree models.
method Analyzed the optimization landscape and convergence of EM algorithm.
result EM algorithm converges to global maximum in latent Gaussian tree models.
Noise-Contrastive Estimation improves efficiency for estimating log-likelihood of complex point processes.
problem Estimating log-likelihood of complex multivariate point processes is computationally expensive.
method Noise-Contrastive Estimation adapted for multivariate point processes, with provable guarantees.
result Our method achieves similar log-likelihood with fewer evaluations and less time.
In this work, a deep learning-based method for log-likelihood ratio (LLR) lossy compression and quantization is proposed, with emphasis on a single-input single-output uncorrelated fading communication setting. A deep autoencoder network is trained to compress, quantize and reconstruct the bit log-likelihood ratios cor…
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 con…
The MAP estimate's log-likelihood sub-optimality is hard to bound in general.
problem Bounding the expected log-likelihood sub-optimality of MAP for exponential families.
method Interpreting MAP as stochastic mirror descent and analyzing convergence rates.
result Current convergence results do not apply to standard examples of exponential families.
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equival…
The log-likelihood loss in heteroscedastic neural networks can lead to poor parameter estimates.
problem Capturing aleatoric uncertainty in deep learning models.
method Examine the log-likelihood loss in conjunction with gradient-based optimizers and propose an alternative formulation, β-NLL. result Using an appropriate β largely mitigates the issue of poor parameter estimates. New method for robust distribution alignment using log-likelihood ratio and normalizing flows.
problem Distribution alignment challenges in deep learning.
method Log-likelihood ratio statistic and normalizing flows.
result Minimizing the proposed objective yields robust domain alignment.
Estimates log-likelihood of interacting particle systems using virtual particles.
problem Inconsistent estimation of finite-particle log-likelihood in large particle systems.
method Stochastic gradient estimate using continuous trajectory and virtual particle systems.
result Convergence to stationary points of limiting mean-field system's log-likelihood.
Study detects signals in spiked Wigner models using log likelihood ratio.
problem Detecting signals in rank-one spiked Wigner models with non-Gaussian noise.
method Proved asymptotic normality of log likelihood ratio and computed error thresholds.
result Optimal signal-to-noise ratio threshold for reliable detection.
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of th…
Clustering, or unsupervised classification, is a task often plagued by outliers. Yet there is a paucity of work on handling outliers in clustering. Outlier identification algorithms tend to fall into three broad categories: outlier inclusion, outlier trimming, and post hoc outlier identification methods, with the forme…
This paper improves SNN training by using multiple sample compartments.
problem Training SNNs with single-sample estimators leads to inaccurate log-likelihood estimates.
method Proposes a GEM-based online learning algorithm that uses multiple independent spiking signals.
result Significant improvements in log-likelihood, accuracy, and calibration with multiple compartments.
Recent work in variational inference (VI) uses ideas from Monte Carlo estimation to tighten the lower bounds on the log-likelihood that are used as objectives. However, there is no systematic understanding of how optimizing different objectives relates to approximating the posterior distribution. Developing such a conn…
New lower bounds improve logistic log-likelihood optimization and inference.
problem Designing computationally tractable lower bounds for logistic log-likelihoods.
method Developed a piece-wise quadratic lower bound that uniformly improves tangent quadratic minorizers.
result Improves the speed of convergence and accuracy of variational Bayes approximations.
This work improves neural network calibration using explicit regularization.
problem Improving predictive uncertainty in neural networks.
method Introducing a probabilistic calibration measure and exploring explicit regularization techniques.
result Explicit regularization improves log-likelihood and predictive uncertainty.
In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from co…
Improved DDPMs achieve high log-likelihoods and sample quality with fewer passes.
problem Improving log-likelihoods of DDPMs while maintaining high sample quality.
method Modifying DDPMs with learning variances and using precision-recall metrics.
result DDPMs can achieve competitive log-likelihoods with fewer forward passes.
This paper addresses the estimation of parameters of a Bayesian network from incomplete data. The task is usually tackled by running the Expectation-Maximization (EM) algorithm several times in order to obtain a high log-likelihood estimate. We argue that choosing the maximum log-likelihood estimate (as well as the max…
New method for online learning in interacting particle systems.
problem Parameter estimation in stochastic interacting particle systems.
method Stochastic approximation of gradient of asymptotic log likelihood using continuous observations.
result Convergence to stationary points of asymptotic log-likelihood under suitable assumptions.
Paper develops unbiased gradient estimator for continuous-time models.
problem Estimating unbiased gradient of log-likelihood for continuous-time models.
method Doubly randomized scheme with coupled conditional particle filter (CCPF).
result Unbiased gradient estimate facilitates gradient-based algorithms.
Separable losses are inconsistent for structured prediction models.
problem Inconsistency of separable losses in structured prediction models.
method Analysis of separable negative log-likelihood losses for structured prediction.
result Separable losses are not Bayes consistent and may not predict the most probable structure.
DNLL loss improves deep LDA accuracy and consistency.
problem Pathological solutions in unconstrained Deep LDA.
method Introducing Discriminative Negative Log-Likelihood (DNLL) loss.
result Deep LDA trained with DNLL produces clean latent spaces and better calibrated probabilities.
Sparse matrices simplify computation of GP variances and likelihoods.
problem Efficient computation of posterior variance and log-likelihood for additive Matérn GPs.
method Represented posterior mean, variance, log-likelihood, and gradient using sparse matrices.
result Efficient computation of posterior mean, variance, log-likelihood, and gradient in O(nlogn) time.