This text explores strategies for learning discrete latent structures in neural networks.
problem Learning discrete latent structures in neural networks is challenging.
method Continuous relaxation, surrogate gradients, and probabilistic estimation.
result Many latent structure learning strategies use the same fundamental building blocks but apply them differently.
NES optimizes discrete structured VAEs effectively without gradient propagation.
problem Learning high-dimensional discrete latent spaces in generative models.
method Natural Evolution Strategies (NES) for gradient-free optimization of discrete structures.
result NES effectively optimizes discrete structured VAEs, comparable to gradient-based methods.
Proposes a VAE with a discrete bottleneck for better text generation.
problem VAEs struggle with latent variable auto-regressive decoding in text generation.
method Introduces a discretized bottleneck to enforce latent feature matching in a compact space.
result Demonstrates improved text generation capabilities across various tasks.
GFlowNet-EM learns complex latent variable models with discrete structures.
problem Challenges in modeling posteriors over discrete compositional latents with expectation-maximization.
method Uses GFlowNets to learn stochastic policies for sampling from complex posterior distributions.
result GFlowNet-EM enables training expressive LVMs with discrete compositional latents.
New spectral clustering method handles discrete covariates for better community detection.
problem Community detection in networks with discrete covariates.
method Spectral algorithm that separates latent network structure from observed covariates.
result Achieves perfect clustering with high probability in large, sparse networks.
GLAD improves latent graph generation by quantizing discrete latent space.
problem Latent space graph generative models lack performance and make unnatural assumptions.
method Adapting diffusion bridges to a discrete latent space, avoiding data space decompositions.
result GLAD achieves competitive performance on graph benchmark datasets.
New method for efficient marginalization of discrete latent variables in neural networks.
problem Computational challenges in training models with discrete latent variables.
method Parameterizing discrete distributions using sparse mappings (sparsemax and structured variants) to reduce support and enable efficient marginalization.
result Achieved good performance in various tasks with efficient and practical training.
DCRL learns causal relationships from mixed-type discrete data.
problem Challenges in learning causal relationships from discrete, mixed-type data.
method Generative framework modeling directed acyclic graph and sparse bipartite graph, flexible measurement models for different types of data.
result Consistent recovery of latent causal structure from observed data distribution.
LADD models improve discrete diffusion for faster language generation.
problem Practical discrete diffusion models ignore cross-token dependencies, degrading performance.
method Introduces a learnable auxiliary latent channel, diffusing over the joint (token, latent) space.
result LADD models yield improvements on unconditional generation metrics.
Generative models with both discrete and continuous latent variables are highly motivated by the structure of many real-world data sets. They present, however, subtleties in training often manifesting in the discrete latent being under leveraged. In this paper, we show that such models are more amenable to training whe…
A new method for generative modeling of discrete data using geometric latent subspaces.
problem Learning generative models for discrete data with statistical dependencies.
method Geometric latent-subspace framework in exponential parameter space of product manifolds of categorical distributions.
result Low-dimensional latent space encodes statistical dependencies and accurately models high-dimensional discrete data.
Unsupervised framework captures acquisition variability in structural connectomes.
problem Acquisition differences across sites, scanners, and protocols complicate structural connectome analysis.
method An unsupervised framework using architectural annealing to balance discrete and continuous latent variables.
result Architectural annealing produces stronger site learning than baseline models.
The paper proposes a method to estimate latent structures in multivariate data without assuming their existence.
problem Estimating latent structures in multivariate distributions that are difficult to identify and reflect the data generating mechanism.
method A model-free approach using a multiscale nonparametric maximum likelihood estimator.
result The method captures meaningful discrete structure at different scales and integrates them to yield an interpretable discrete representation.
In this work we present Discrete Attend Infer Repeat (Discrete-AIR), a Recurrent Auto-Encoder with structured latent distributions containing discrete categorical distributions, continuous attribute distributions, and factorised spatial attention. While inspired by the original AIR model andretaining AIR model's capabi…
New method learns disentangled discrete representations using categorical variational autoencoders.
problem Learning disentangled representations from discrete latent spaces.
method Replaced standard Gaussian VAE with a categorical VAE to mitigate rotational invariance.
result Categorical distributions improve learning of disentangled representations.
Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for m…
Reparameterization of variational auto-encoders with continuous random variables is an effective method for reducing the variance of their gradient estimates. In the discrete case, one can perform reparametrization using the Gumbel-Max trick, but the resulting objective relies on an argmax operation and is non-dif…
New JSA autoencoders tackle discrete latent variable models for semi-supervised learning.
problem Handling discrete observations and latent codes in deep generative models.
method Joint-stochastic-approximation (JSA) autoencoders that directly maximize data log-likelihood and minimize KL divergence.
result JSA autoencoders achieve comparable performance to continuous latent space models in semi-supervised tasks.
A new model for time series using discrete latent states.
problem Efficiently modeling time series data with discrete latent states.
method A Markov chain-based model for training high-dimensional discrete latent data.
result Improved performance on time series datasets.
We present a semi-supervised learning algorithm for learning discrete factor analysis models with arbitrary structure on the latent variables. Our algorithm assumes that every latent variable has an "anchor", an observed variable with only that latent variable as its parent. Given such anchors, we show that it is possi…
Structured latent attribute models (SLAMs) are a special family of discrete latent variable models widely used in social and biological sciences. This paper considers the problem of learning significant attribute patterns from a SLAM with potentially high-dimensional configurations of the latent attributes. We address …
PRISM-VQ combines financial priors with vector quantization for better stock prediction.
problem Predicting cross-sectional stock returns is hard due to low signal-to-noise ratios and changing market conditions.
method Integrates expert priors, vector-quantized latent factors, and dynamic factor loadings.
result Consistent improvements in cross-sectional return prediction and portfolio performance.
Bayesian active learning reduces data needed for latent variable models.
problem Active learning for latent variable models, especially mixtures of linear regressions and HMMs with GLM observations.
method Maximum-mutual-information input selection for discrete latent variable regression models.
result Active learning can achieve large gains for mixtures of linear-Gaussian models and substantially reduces data needed for GLM-HMM.
New method uses joint stochastic approximation to improve learning of discrete latent models.
problem Challenges in learning discrete latent variable models, especially with inference model gradients and log-likelihood optimization.
method Proposes a new method based on stochastic approximation theory that directly maximizes the target log-likelihood and minimizes the posterior-inference model divergence.
result Consistently outperforms recent competitive algorithms in generative modeling and structured prediction tasks.
Paper learns latent and hierarchical structures in CDMs from data.
problem Jointly learning latent and hierarchical structures in CDMs from observed data.
method Penalized likelihood approach for selecting attributes and estimating structures; EM and latent structure recovery algorithms.
result Good performance demonstrated by simulation and real data applications.
Paper uncovers causal structures in Hawkes processes with latent subprocesses.
problem Tackles latent subprocesses in Hawkes processes with complex event-driven interactions.
method Proposes a two-phase iterative algorithm that infers causal relationships and identifies latent subprocesses.
result Successfully recovers causal structures in datasets with latent subprocesses.
Formalizes concepts as latent variables in hierarchical models for high-dimensional data.
problem Lack of formalization and theoretical insights for learning discrete concepts from high-dimensional data.
method Formalizes concepts as latent causal variables in a hierarchical model, formulates conditions for concept identification.
result Conditions for identifying latent hierarchical models in unsupervised data, handling complex structures and high-dimensional data.
Direct optimization of binary latent VAEs achieves competitive results without sampling.
problem Training VAEs with discrete latent variables using standard methods is challenging.
method Applied evolutionary algorithms to directly optimize discrete latent distributions.
result Direct optimization is efficient and competitive in zero-shot learning.
Improved hierarchical discrete VAEs for better stability and performance.
problem Training stable and efficient hierarchical discrete VAEs with numerous latent variables.
method Introducing Relaxed-Responsibility Vector-Quantisation to parameterise discrete latent variables in a hierarchical structure.
result Achieved state-of-the-art bits-per-dim results for various standard datasets.
CADD improves generative quality by augmenting discrete diffusion with continuous latent space.
problem Loss of semantic information between denoising steps in discrete diffusion models.
method Introduces a framework that augments discrete state space with a continuous latent space, allowing for graded, informative masked tokens.
result CADD improves generative quality across text generation, image synthesis, and code modeling.
Temporal Point Processes (TPP) with partial likelihoods involving a latent structure often entail an intractable marginalization, thus making inference hard. We propose a novel approach to Maximum Likelihood Estimation (MLE) involving approximate inference over the latent variables by minimizing a tight upper bound on …
Latent Dirichlet Allocation models discrete data as a mixture of discrete distributions, using Dirichlet beliefs over the mixture weights. We study a variation of this concept, in which the documents' mixture weight beliefs are replaced with squashed Gaussian distributions. This allows documents to be associated with e…
A new method reduces variance in training discrete latent variable models.
problem High variance in stochastic gradient estimators for discrete latent variable models.
method Double control variates for score function estimators using Taylor expansions.
result Our method can have lower variance compared to other estimators.
Deep Discrete Encoders (DDEs) tackle interpretable generative models for rich data with discrete latent layers.
problem Overparametrized, non-identifiable, and uninterpretable deep generative models in high-stakes applications.
method Directed graphical model with multiple binary latent layers, transparent identifiability conditions, scalable estimation pipeline.
result Transparent identifiability conditions and scalable estimation pipeline for interpretable DDEs.
Proposes a non-parametric method for deep discrete latent variable models.
problem Learning sparse discrete latent representations in deep models.
method Iterative algorithm with Beta-Bernoulli process prior and local data scaling.
result Improves sparsity and scalability of deep discrete latent variable models.
We present the Infinite Latent Events Model, a nonparametric hierarchical Bayesian distribution over infinite dimensional Dynamic Bayesian Networks with binary state representations and noisy-OR-like transitions. The distribution can be used to learn structure in discrete timeseries data by simultaneously inferring a s…
LDTA expands LDA's topic modeling capacity with tree-structured priors.
problem Limited expressiveness of Dirichlet priors in LDA for complex topic relationships.
method Introduces Latent Dirichlet-Tree Allocation (LDTA) with Dirichlet-Tree (DT) priors, and develops universal mean-field variational inference and Expectation Propagation.
result LDTA enables expressive, tree-structured priors over topic proportions, expanding modeling capacity of LDA.
Graphical models are commonly used tools for modeling multivariate random variables. While there exist many convenient multivariate distributions such as Gaussian distribution for continuous data, mixed data with the presence of discrete variables or a combination of both continuous and discrete variables poses new cha…
Latent feature modeling allows capturing the latent structure responsible for generating the observed properties of a set of objects. It is often used to make predictions either for new values of interest or missing information in the original data, as well as to perform data exploratory analysis. However, although the…
Probabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete late…
New estimators reduce variance in training variational autoencoders with discrete latent variables.
problem Training variational autoencoders with discrete latent variables requires efficient gradient estimation.
method Introduce ReinMax-Rao and ReinMax-CV estimators using Rao-Blackwellisation and control variates.
result Demonstrate superior performance on training variational autoencoders with discrete latent spaces.
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
In this paper we present a fully Bayesian latent variable model which exploits conditional nonlinear(in)-dependence structures to learn an efficient latent representation. The latent space is factorized to represent shared and private information from multiple views of the data. In contrast to previous approaches, we i…
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary lat…
LOL-BO improves latent space Bayesian optimization over structured inputs.
problem Optimizing complex functions over high-dimensional, structured search spaces.
method Adapting trust regions from high-dimensional to structured settings, using a DAE to map inputs into a latent space.
result Achieves up to 20x improvement over state-of-the-art methods.
Paper proposes a new generative model for discrete distributions using flows on submanifolds.
problem Discretization issues and complex statistical dependencies in discrete data.
method Continuous normalizing flows on factorizing discrete measures, geodesic flow matching.
result Efficient training and broad applicability demonstrated through experiments.
This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness. The PRGDS is based on a new motif in Bayesian latent variable modeling, an alternating chain of discrete Poisson and continuous …
Proposes a hierarchical model for learning discrete Bayesian networks with shrinkage.
problem Learning discrete Bayesian networks with high-order interactions and cell probabilities.
method Hierarchical Dirichlet shrinkage model with Metropolis-adjusted Langevin algorithm for sampling.
result Efficiently learns graph structure and selects between DAGs from sparse count data.