Deep Discrete Encoders (DDEs) tackle interpretable generative models for rich data with discrete latent layers.
problem Overparametrized, non-identifiable, and uninterpretable deep generative models in high-stakes applications.
method Directed graphical model with multiple binary latent layers, transparent identifiability conditions, scalable estimation pipeline.
result Transparent identifiability conditions and scalable estimation pipeline for interpretable DDEs.
Improved hierarchical discrete VAEs for better stability and performance.
problem Training stable and efficient hierarchical discrete VAEs with numerous latent variables.
method Introducing Relaxed-Responsibility Vector-Quantisation to parameterise discrete latent variables in a hierarchical structure.
result Achieved state-of-the-art bits-per-dim results for various standard datasets.
It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently …
DCRL learns causal relationships from mixed-type discrete data.
problem Challenges in learning causal relationships from discrete, mixed-type data.
method Generative framework modeling directed acyclic graph and sparse bipartite graph, flexible measurement models for different types of data.
result Consistent recovery of latent causal structure from observed data distribution.
NegBio-VAE models neural spike counts with negative binomial distribution.
problem Limited biological plausibility of continuous latent variables in VAEs for neural spike modeling.
method Proposes a negative binomial latent-variable model with a dispersion parameter for overdispersed spike count modeling.
result NegBio-VAE outperforms competing models in reconstruction and generation tasks.
Identifies interpretable generative model for multivariate data.
problem Black-box architectures of deep generative models are often unidentified and difficult to interpret.
method Introduces Deep Discrete Encoder (DDE) Copula, a hierarchical binary latent variable model inside a copula framework.
result Establishes conditions for identification of DDE copula parameters and proves posterior consistency.
We introduce a new framework for unsupervised learning of representations based on a novel hierarchical decomposition of information. Intuitively, data is passed through a series of progressively fine-grained sieves. Each layer of the sieve recovers a single latent factor that is maximally informative about multivariat…
New model for multi-layer categorical data improves latent class analysis.
problem Traditional latent class analysis for single-layer categorical data is insufficient for multi-layer data.
method Developed a multi-layer latent class model (multi-layer LCM) and three spectral methods for estimation.
result The debiased sum of Gram matrices method performs best in estimating latent classes.
LADD models improve discrete diffusion for faster language generation.
problem Practical discrete diffusion models ignore cross-token dependencies, degrading performance.
method Introduces a learnable auxiliary latent channel, diffusing over the joint (token, latent) space.
result LADD models yield improvements on unconditional generation metrics.
A new model for time series using discrete latent states.
problem Efficiently modeling time series data with discrete latent states.
method A Markov chain-based model for training high-dimensional discrete latent data.
result Improved performance on time series datasets.
Proposes a VAE with a discrete bottleneck for better text generation.
problem VAEs struggle with latent variable auto-regressive decoding in text generation.
method Introduces a discretized bottleneck to enforce latent feature matching in a compact space.
result Demonstrates improved text generation capabilities across various tasks.
GFlowNet-EM learns complex latent variable models with discrete structures.
problem Challenges in modeling posteriors over discrete compositional latents with expectation-maximization.
method Uses GFlowNets to learn stochastic policies for sampling from complex posterior distributions.
result GFlowNet-EM enables training expressive LVMs with discrete compositional latents.
We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acq…
A new method for generative modeling of discrete data using geometric latent subspaces.
problem Learning generative models for discrete data with statistical dependencies.
method Geometric latent-subspace framework in exponential parameter space of product manifolds of categorical distributions.
result Low-dimensional latent space encodes statistical dependencies and accurately models high-dimensional discrete data.
Generative models with both discrete and continuous latent variables are highly motivated by the structure of many real-world data sets. They present, however, subtleties in training often manifesting in the discrete latent being under leveraged. In this paper, we show that such models are more amenable to training whe…
Direct optimization of binary latent VAEs achieves competitive results without sampling.
problem Training VAEs with discrete latent variables using standard methods is challenging.
method Applied evolutionary algorithms to directly optimize discrete latent distributions.
result Direct optimization is efficient and competitive in zero-shot learning.
GLAD improves latent graph generation by quantizing discrete latent space.
problem Latent space graph generative models lack performance and make unnatural assumptions.
method Adapting diffusion bridges to a discrete latent space, avoiding data space decompositions.
result GLAD achieves competitive performance on graph benchmark datasets.
CADD improves generative quality by augmenting discrete diffusion with continuous latent space.
problem Loss of semantic information between denoising steps in discrete diffusion models.
method Introduces a framework that augments discrete state space with a continuous latent space, allowing for graded, informative masked tokens.
result CADD improves generative quality across text generation, image synthesis, and code modeling.
This text explores strategies for learning discrete latent structures in neural networks.
problem Learning discrete latent structures in neural networks is challenging.
method Continuous relaxation, surrogate gradients, and probabilistic estimation.
result Many latent structure learning strategies use the same fundamental building blocks but apply them differently.
Physics-informed neural operator learns from coarse to fine discretized data.
problem Lack of high-fidelity training data and uneven grid resolution.
method Physics-informed multi-resolution neural operator framework.
result Learn from arbitrarily discretized input functions using latent embedding and finite difference solver.
A new method reduces variance in training discrete latent variable models.
problem High variance in stochastic gradient estimators for discrete latent variable models.
method Double control variates for score function estimators using Taylor expansions.
result Our method can have lower variance compared to other estimators.
Quantitatively assessing relationships between latent variables and observed variables is important for understanding and developing generative models and representation learning. In this paper, we propose latent-observed dissimilarity (LOD) to evaluate the dissimilarity between the probabilistic characteristics of lat…
New method for efficient marginalization of discrete latent variables in neural networks.
problem Computational challenges in training models with discrete latent variables.
method Parameterizing discrete distributions using sparse mappings (sparsemax and structured variants) to reduce support and enable efficient marginalization.
result Achieved good performance in various tasks with efficient and practical training.
NES optimizes discrete structured VAEs effectively without gradient propagation.
problem Learning high-dimensional discrete latent spaces in generative models.
method Natural Evolution Strategies (NES) for gradient-free optimization of discrete structures.
result NES effectively optimizes discrete structured VAEs, comparable to gradient-based methods.
Proposes a non-parametric method for deep discrete latent variable models.
problem Learning sparse discrete latent representations in deep models.
method Iterative algorithm with Beta-Bernoulli process prior and local data scaling.
result Improves sparsity and scalability of deep discrete latent variable models.
Probabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete late…
New estimators reduce variance in training variational autoencoders with discrete latent variables.
problem Training variational autoencoders with discrete latent variables requires efficient gradient estimation.
method Introduce ReinMax-Rao and ReinMax-CV estimators using Rao-Blackwellisation and control variates.
result Demonstrate superior performance on training variational autoencoders with discrete latent spaces.
Neural networks are vulnerable to adversarial attacks -- small visually imperceptible crafted noise which when added to the input drastically changes the output. The most effective method of defending against these adversarial attacks is to use the methodology of adversarial training. We analyze the adversarially train…
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
New spectral clustering method handles discrete covariates for better community detection.
problem Community detection in networks with discrete covariates.
method Spectral algorithm that separates latent network structure from observed covariates.
result Achieves perfect clustering with high probability in large, sparse networks.
New JSA autoencoders tackle discrete latent variable models for semi-supervised learning.
problem Handling discrete observations and latent codes in deep generative models.
method Joint-stochastic-approximation (JSA) autoencoders that directly maximize data log-likelihood and minimize KL divergence.
result JSA autoencoders achieve comparable performance to continuous latent space models in semi-supervised tasks.
Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary lat…
To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reducti…
In this work we present Discrete Attend Infer Repeat (Discrete-AIR), a Recurrent Auto-Encoder with structured latent distributions containing discrete categorical distributions, continuous attribute distributions, and factorised spatial attention. While inspired by the original AIR model andretaining AIR model's capabi…
Multilayer switch networks are proposed as artificial generators of high-dimensional discrete data (e.g., binary vectors, categorical data, natural language, network log files, and discrete-valued time series). Unlike deconvolution networks which generate continuous-valued data and which consist of upsampling filters a…
Paper introduces TSSDMN for modeling dynamic multilayer networks.
problem Capturing temporal and cross-layer dynamics in multilayer networks.
method Tensor State Space Model (TSSDMN) using symmetric Tucker decomposition.
result TSSDMN uniquely captures temporal dynamics within and across layers.
Framework estimates multiple plausible solutions with uncertainty measures.
problem Machine learning models need to propose multiple plausible solutions with meaningful uncertainty.
method Discrete latent variables model one-to-many mappings, allowing effective conditional probability estimation.
result Framework outperforms state-of-the-art in uncertainty estimation and is practical.
New method learns disentangled discrete representations using categorical variational autoencoders.
problem Learning disentangled representations from discrete latent spaces.
method Replaced standard Gaussian VAE with a categorical VAE to mitigate rotational invariance.
result Categorical distributions improve learning of disentangled representations.
We present a framework for learning disentangled and interpretable jointly continuous and discrete representations in an unsupervised manner. By augmenting the continuous latent distribution of variational autoencoders with a relaxed discrete distribution and controlling the amount of information encoded in each latent…
Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distri…
What does a neural network encode about a concept as we traverse through the layers? Interpretability in machine learning is undoubtedly important, but the calculations of neural networks are very challenging to understand. Attempts to see inside their hidden layers can either be misleading, unusable, or rely on the la…
Paper optimizes clustering for multi-layer networks and discrete mixtures.
problem Optimizing clustering in multi-layer networks and discrete mixtures.
method Two-stage method: tensor-based initialization and likelihood-based refinement.
result Achieves minimax optimal error rate for multi-layer networks and discrete mixtures.
Consider a quantum particle trapped between a curved layer of constant width built over a complete, non-compact, C2 smooth surface embedded in R3. We assume that the surface is asymptotically flat in the sense that the second fundamental form vanishes at infinity, and that the surface is not tot…
DD-VAE uses deterministic decoding for better latent code utilization in discrete data.
problem Inflexible decoders in VAEs lead to poor utilization of latent codes in discrete data.
method Proposed DD-VAE with deterministic decoding and new proposal distributions.
result DD-VAE improves latent code utilization and structure of learned manifold.
Improved training for VQ-VAE models with robust codebook learning.
problem Challenges in training discrete latent variable models, especially VQ-VAEs.
method Increased learning rate and periodic re-initialization of codebook for robust training.
result More robust training and increased usage of latent codewords, even for large codebooks.
The emergence of data-driven demand analysis has led to the increased use of generative modelling to learn the probabilistic dependencies between random variables. Although their apparent use has mostly been limited to image recognition and classification in recent years, generative machine learning algorithms can be a…
A new method learns hierarchical EBM models with diffusion schemes.
problem Challenges in learning EBM models with multi-modal distributions.
method Proposes a diffusion probabilistic scheme to learn EBM models in hierarchical latent spaces.
result Demonstrates superior performance on various tasks with diffusion-learned EBM.
We propose a neural network for unsupervised anomaly detection with a novel robust subspace recovery layer (RSR layer). This layer seeks to extract the underlying subspace from a latent representation of the given data and removes outliers that lie away from this subspace. It is used within an autoencoder. The encoder …