New method resolves nonidentifiability in mixture models.
problem Nonidentifiability in marginal models of mixture models.
method Introducing an effective temperature to generalize the marginal likelihood.
result Maximization of the generalized likelihood leads to unique results.
We consider two connected aspects of maximum likelihood estimation of the parameter for high-dimensional discrete graphical models: the existence of the maximum likelihood estimate (mle) and its computation. When the data is sparse, there are many zeros in the contingency table and the maximum likelihood estimate of th…
New algorithm improves latent variable model estimation.
problem Estimating parameters in latent variable models.
method Jarzynski-adjusted Langevin algorithm (JALA) for SMC methods.
result JALA-EM provides maximum marginal likelihood estimate.
Graphical models with bi-directed edges (<->) represent marginal independence: the absence of an edge between two vertices indicates that the corresponding variables are marginally independent. In this paper, we consider maximum likelihood estimation in the case of continuous variables with a Gaussian joint distributio…
Supervised topic models utilize document's side information for discovering predictive low dimensional representations of documents. Existing models apply the likelihood-based estimation. In this paper, we present a general framework of max-margin supervised topic models for both continuous and categorical response var…
Combines RL and MML to learn programs from indirect supervision.
problem Learning programs from indirect supervision without spurious solutions.
method Connects RL and MML, uses systematic search and randomized exploration.
result Significant gains over state-of-the-art semantic parsers.
We define a generalized likelihood function based on uncertainty measures and show that maximizing such a likelihood function for different measures induces different types of classifiers. In the probabilistic framework, we obtain classifiers that optimize the cross-entropy function. In the possibilistic framework, we …
We consider discrete graphical models Markov with respect to a graph G and propose two distributed marginal methods to estimate the maximum likelihood estimate of the canonical parameter of the model. Both methods are based on a relaxation of the marginal likelihood obtained by considering the density of the variable…
Bayesian networks with latent variables are characterized and their likelihoods compared.
problem Characterizing and comparing likelihoods of Bayesian networks with latent variables.
method Characterized likelihood function and empirical Bayesian network. Proved dominance of global maximum likelihood from empirical model.
result The global maximum likelihood of the original Bayesian network is attained if and only if parameters are consistent with empirical model.
We give polynomial-time algorithms for the exact computation of lowest-energy (ground) states, worst margin violators, log partition functions, and marginal edge probabilities in certain binary undirected graphical models. Our approach provides an interesting alternative to the well-known graph cut paradigm in that it …
Improved Gaussian process regression with tighter log marginal likelihood bounds.
problem Improving predictive performance in Gaussian process regression models.
method Lower bound on log marginal likelihood using conjugate gradients.
result Improved predictive performance compared to other conjugate gradient based approaches.
A teacher can improve a learner's performance by selecting a smaller, more effective training subset.
problem Improving a learner's performance by selecting a smaller training subset.
method Sharp guarantees for two learners and a mixed-integer nonlinear programming-based algorithm for general learners.
result Empirical experiments show that the algorithm finds good super-teaching sets for regression and classification problems.
New method learns latent energy models using particle algorithms.
problem Learning latent variable models with energy priors.
method Continuous-time SDEs for MMLE, particle-based discretization.
result Practical algorithm converges to solve MMLE problem.
New algorithms learn latent variable models without tuning, outperforming existing methods.
problem Learning latent variable models without manual tuning.
method Two particle-based algorithms using free energy minimization and coin betting.
result Learning algorithms are entirely tuning-free and competitive with existing methods.
We consider distributed estimation of the inverse covariance matrix, also called the concentration or precision matrix, in Gaussian graphical models. Traditional centralized estimation often requires global inference of the covariance matrix, which can be computationally intensive in large dimensions. Approximate infer…
The normalized maximized likelihood (NML) provides the minimax regret solution in universal data compression, gambling, and prediction, and it plays an essential role in the minimum description length (MDL) method of statistical modeling and estimation. Here we show that the normalized maximum likelihood has a Bayes-li…
We simplify inference for TPP models with latent structures.
problem Intractable marginalization in TPP models with latent structures.
method Approximate inference over latent variables using a tight upper bound on the approximation gap.
result Improved results for models like Survival Analysis.
Ultrahigh-dimensional variable selection plays an increasingly important role in contemporary scientific discoveries and statistical research. Among others, Fan and Lv [J. R. Stat. Soc. Ser. B Stat. Methodol. 70 (2008) 849-911] propose an independent screening framework by ranking the marginal correlations. They showed…
Develops a new algorithm for estimating model parameters using interacting particle systems.
problem Estimating parameters of latent variable models.
method Interacting Particle Langevin Algorithm (IPLA) based on Langevin diffusion.
result Nonasymptotic optimisation error bounds for the estimator.
A scalable method for estimating spatial data using VREML.
problem Costly computation of REML for large, sparse precision matrices in spatial data.
method Proposes VREML framework approximating marginal likelihood with Gaussian variational distribution and deriving a coordinate-ascent algorithm.
result Empirically shows VREML outperforms MLE and INLA.
New CTRL algorithm adapts to varying problem difficulty.
problem Adapting to varying levels of problem difficulty in CTRL.
method MLE with a general function approximator, estimating state marginal density.
result Regret bound scales with reward variance and measurement resolution, independent of measurement strategy.
Identifies interpretable generative model for multivariate data.
problem Black-box architectures of deep generative models are often unidentified and difficult to interpret.
method Introduces Deep Discrete Encoder (DDE) Copula, a hierarchical binary latent variable model inside a copula framework.
result Establishes conditions for identification of DDE copula parameters and proves posterior consistency.
Graphical models trained using maximum likelihood are a common tool for probabilistic inference of marginal distributions. However, this approach suffers difficulties when either the inference process or the model is approximate. In this paper, the inference process is first defined to be the minimization of a convex f…
New SMC samplers improve stochastic optimisation efficiency.
problem Optimizing functions with intractable gradients in machine learning and statistics.
method Sequential Monte Carlo (SMC) samplers for stochastic optimisation.
result Significant computational gains achieved with SMC approximations.
New insights into using IPF for inferring dynamic networks from marginals.
problem Inferring dynamic networks from time-aggregated adjacency matrices and time-varying marginals.
method Identifying a generative network model and establishing its maximum likelihood estimates via IPF, with convergence guarantees for sparse data.
result IPF provides principled estimation of dynamic networks from marginals under certain conditions, with structure-dependent error bounds and guaranteed convergence for sparse data.
Accelerates MMLE using SVGD with Nesterov acceleration.
problem Maximum Marginal Likelihood Estimation optimization.
method Stein variational gradient descent with Nesterov acceleration.
result Consistently accelerates convergence across various tasks.
Variational Prediction simplifies Bayesian inference without test time costs.
problem Bayesian inference's computational costs and posterior predictive distribution marginalization.
method Variational Prediction learns a variational approximation to the posterior predictive distribution using a variational bound.
result Directly learns a variational approximation to the posterior predictive distribution without test time marginalization costs.
New insights into Gamma-Poisson model for count data.
problem Estimating topic/dictionary matrix robustness to rank over-specification.
method Rewriting GaP model free of score/activation matrix, leading to new MME algorithm.
result Automatic pruning of irrelevant dictionary columns observed empirically.
Maximum likelihood estimator performance in logistic regression analyzed.
problem Performance of maximum likelihood estimator in logistic regression.
method Sharp non-asymptotic guarantees for existence and excess logistic risk.
result Sharp guarantees for the existence and excess risk of MLE in logistic regression.
A new EM-based algorithm improves deep generative model training.
problem Training deep generative models with maximum likelihood is challenging.
method The paper proposes reweighted expectation maximization (REM), a new algorithm that directly maximizes the log marginal likelihood of the data.
result REM learns better generative models than the IWAE, leading to significantly better performance in density estimation benchmarks.
This research improves neural network representation identifiability through task structures.
problem Improving neural network representation identifiability in multi-task settings.
method Analyzing the effects of task distributions and causal structures on latent factors, leading to simpler optimization.
result A straightforward optimization procedure enables better representation recovery in both synthetic and real-world data.
A new method estimates marginal likelihood using normalizing flows.
problem Estimating marginal likelihood in Bayesian model selection.
method Learned harmonic mean estimator using normalizing flows.
result Normalizing flows avoid the exploding variance problem.
Linear VAEs explain posterior collapse in VAEs via local maxima in log marginal likelihood.
problem Posterior collapse in VAEs where variational posterior matches prior for some latent variables.
method Analysis of linear VAEs and their relation to pPCA, proving ELBO does not introduce spurious local maxima.
result Linear VAEs have identifiable global maxima corresponding to principal component directions, explaining posterior collapse.
Unbiased gradient estimation improves VAE performance.
problem Training VAEs via maximum likelihood is difficult due to intractable integrals.
method Introduced unbiased estimators of the log-likelihood gradient using coupled Markov chains.
result Unbiased estimators lead to better predictive performance in VAEs.
The paper analyzes SBL pruning criteria under weakened assumptions.
problem Sparse Bayesian learning hyperparameter divergence and pruning.
method Analyzing marginal likelihood function under weakened Gaussian assumptions.
result Conditions for finite vs infinite hyperparameters lead to F-SBL pruning.
Paper proposes a new method to predict partial rankings from crowdsourced data.
problem Ambiguity in pairwise comparisons leads to incomplete rankings, requiring a better method.
method Margin-based Maximum Likelihood Estimate (MLE) framework for probabilistic partial order learning.
result The proposed MLE method improves accuracy over traditional algorithms.
Sublinear time DPPs model text documents efficiently.
problem Efficiently modeling large sets of items with DPPs.
method Low-rank factorization of marginal kernel for sublinear time manipulation.
result Conditional maximum likelihood for topic proportions without approximation.
Paper shows how sparse inversion speeds up log determinant derivatives.
problem Deriving log determinant derivatives for sparse matrices.
method Sparse inversion, selected inversion, accelerates computation.
result Derivative of log determinant can be computed faster with sparse inversion.
Improved BP on CRBMs yields better structured prediction results.
problem Structured prediction on CRBMs using current methods is suboptimal.
method Matrix-based belief propagation implementation for CRBMs.
result BP on CRBMs provides significantly better results than CD methods.
Efficiently marginalizes over Gaussian Process kernels for better model flexibility and uncertainty.
problem Inefficient marginalization over Gaussian Process kernels for large datasets.
method Bayesian Quadrature scheme with maximum mean discrepancies and invariances between Spectral Mixture kernels.
result Achieves more accurate predictions and better calibrated uncertainty than state-of-the-art baselines.
A new method models volatile financial time series using v-transforms and copulas.
problem Modeling volatile financial time series with standard methods.
method v-transforms and copulas to describe and estimate time series with arbitrary marginal distributions and copula dynamics.
result The model replicates stylized facts of financial return series and facilitates risk quantification.
We address the problem of learning the parameters in graphical models when inference is intractable. A common strategy in this case is to replace the partition function with its Bethe approximation. We show that there exists a regime of empirical marginals where such Bethe learning will fail. By failure we mean that th…
Estimates log marginal likelihood using multilevel Monte Carlo.
problem Estimating log marginal likelihood accurately.
method Unbiased multilevel Monte Carlo estimator.
result Validates application in variational Bayes.
Optimization results are one method for understanding neural computation from Nature's perspective and for defining the physical limits on neuron-like engineering. Earlier work looks at individual properties or performance criteria and occasionally a combination of two, such as energy and information. Here we make use …
A new copula estimation method using classification.
problem Estimating copula density from joint and marginal distributions.
method Train a classifier to distinguish joint density from product of marginals.
result Empirically outperforms existing copula estimators.
Proposes a new method for high-dimensional density estimation.
problem Estimating high-dimensional probability density functions efficiently.
method Tensorizing flow method combining tensor-train and flow-based generative modeling.
result Efficiently constructs an approximate density in tensor-train form and trains a flow model to match empirical distribution.
Bayesian evidence helps compare models but can overfit.
problem Comparing hypotheses consistent with observations.
method Marginal likelihood, Occam's razor, PAC-Bayes bounds.
result Marginal likelihood can negatively correlate with generalization.
Efficiently estimates marginal likelihood using SGAIS.
problem Estimating marginal likelihood in i.i.d. data settings.
method Stochastic Gradient Annealed Importance Sampling (SGAIS).
result Significantly faster and more accurate estimates of marginal likelihood.