Paper shows how SFA fits into FBM framework for time series separation.
problem Identifying time series decomposition in flow-based models.
method Combining SFA and FBM to make time series decomposition identifiable.
result Time series decomposition becomes identifiable using SFA and FBM.
The paper explores how invertibility affects the complexity of encoder models in VAEs.
problem The complexity of the encoder model in VAEs when the generative map is invertible.
method Formalizes the concept of strong invertibility and analyzes the complexity of the encoder model.
result Strongly invertible generative maps allow for simpler encoder models, while non-invertible maps require exponentially larger encoders.
Study nonparametric factor analysis with arbitrary noise.
problem Identify latent variables in noisy, non-invertible settings.
method Developed a general framework and estimation methods.
result Identify latent variables up to certain indeterminacies.
In many tasks, in particular in natural science, the goal is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter- to measurement-space is a well-defined function, whereas the inverse problem is ambiguous: one measurement may map to multiple different sets of param…
Improved sampling quality with RBM-Flow and D-Flow models.
problem Efficient sampling of complex data distributions using invertible flows.
method Implement RBM-Flow and D-Flow models with discrete latent variables.
result Significant improvement in sampling quality over baseline models.
The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed f…
A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative proc…
We propose two neural network based mixture models in this article. The proposed mixture models are explicit in nature. The explicit models have analytical forms with the advantages of computing likelihood and efficiency of generating samples. Computation of likelihood is an important aspect of our models. Expectation-…
The paper stabilizes invertible neural networks by using Gaussian mixture models.
problem Invertible neural networks can have exploding Lipschitz constants, leading to numerical errors.
method The authors use Gaussian mixture models to stabilize the latent distribution of invertible neural networks.
result Numerical simulations confirm that this modification improves sampling quality in multimodal applications.
Researchers identify latent variables and causal structures from nonlinear hierarchical models.
problem Challenging task of identifying latent variables and causal structures from observational data, especially when relationships are nonlinear.
method Investigated nonlinear latent hierarchical causal models, developed identification criterion, and constructed an estimation procedure.
result Identifiability of causal structures and latent variables achieved under mild assumptions.
Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transf…
Stochastic gradient Langevin dynamics (SGLD) is a computationally efficient sampler for Bayesian posterior inference given a large scale dataset. Although SGLD is designed for unbounded random variables, many practical models incorporate variables with boundaries such as non-negative ones or those in a finite interval.…
Study identifies latent variables and models from spacecraft data.
problem Learning reliable models from spacecraft data with complex relationships.
method Inductive bias inspired by controllable canonical forms for sparse, input-dependent latent variables.
result Identifies latent variables up to scaling and determines dynamic models up to transformations for linear and affine systems.
Here, we present a novel approach to solve the problem of reconstructing perceived stimuli from brain responses by combining probabilistic inference with deep learning. Our approach first inverts the linear transformation from latent features to brain responses with maximum a posteriori estimation and then inverts the …
A neural network finds causal relationships among latent variables.
problem Learning causal structure among latent variables in high-dimensional data.
method Redundant Input Neural Network (RINN) with modified architecture and regularized objective function.
result The RINN method successfully recovers latent causal structure between input and output variables.
New method identifies latent variables in cognitive models using neural networks.
problem Inference of latent variables in complex cognitive models is limited.
method Recurrent neural networks and simulation-based inference for latent variable sequences.
result Extends neural Bayes estimation to broader classes of cognitive models.
DSVNP uses global and local latent variables for improved neural process predictions.
problem Limited expressiveness of vanilla neural processes in capturing target-specific local variation.
method Introduces DSVNP combining global and local latent variables for prediction.
result Competitive prediction performance in multi-output regression and uncertainty estimation.
We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acq…
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
problem Exploding inverses in INNs cause numerical non-invertibility, leading to failures in various tasks.
method Derived bi-Lipschitz properties of INN building blocks, proposed regularizers for local invertibility, and stable INN designs for global invertibility.
result Bi-Lipschitz properties and stable INN designs are crucial for addressing numerical non-invertibility.
LEAP identifies latent causal variables from temporal data.
problem Recovering time-delayed latent causal variables from general temporal data.
method Proposes LEAP, a framework that extends VAEs with constraints for temporally causal latent processes.
result Successfully identifies temporally causal latent processes from observed variables under various dependency structures.
New method identifies latent variables with sparse perturbations.
problem Identifying latent variables with minimal supervision.
method Weakly supervised representation learning with sparse perturbations.
result Identification of latent variables up to specified blocks.
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 con…
Inferring the latent variable generating a given test sample is a challenging problem in Generative Adversarial Networks (GANs). In this paper, we propose InvGAN - a novel framework for solving the inference problem in GANs, which involves training an encoder network capable of inverting a pre-trained generator network…
New model uncovers non-Euclidean neural representations.
problem Discovering latent neural states in complex, non-Euclidean spaces.
method Manifold GPLVM for identifying latent variables and neural contributions.
result mGPLVM correctly recovers non-Euclidean latent structures in neural data.
Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for m…
Improved robust latent variable estimation for neural dynamics.
problem Inconsistent results due to noise and nonlinearity in existing models.
method Probabilistic approach to latent variable estimation in decomposed models.
result More accurate latent variable inference in nonlinear systems with diverse noise conditions.
CIPNN model tackles continuous latent variables, solving intractable posterior problems.
problem Solving intractable posterior calculation for continuous latent variables.
method Derives analytical solution for posterior of continuous latent variables, proposes CIPNN and CIPAE.
result CIPNN model demonstrates great classification capability, solving problems for continuous latent variables.
Normalising flows (NFS) map two density functions via a differentiable bijection whose Jacobian determinant can be computed efficiently. Recently, as an alternative to hand-crafted bijections, Huang et al. (2018) proposed neural autoregressive flow (NAF) which is a universal approximator for density functions. Their fl…
Variational autoencoders often collapse, showing latent variables are non-identifiable.
problem Posterior collapse in variational autoencoders due to non-identifiable latent variables.
method Proves latent variable non-identifiability causes posterior collapse. Proposes latent-identifiable models using Brenier maps and input convex neural networks.
result Latent-identifiable models resolve posterior collapse and provide meaningful representations.
This paper shows cross-entropy can recover latent structures in supervised learning.
problem Understanding why supervised learning works well and how models learn interpretable factors of variation.
method Extending identifiability results to parametric instance discrimination, proving cross-entropy minimization can recover latent structures up to linear transformations.
result Models trained with cross-entropy can learn representations of ground-truth factors of variation up to a linear transformation.
pi-VAE models neural activity with interpretable latent variables.
problem Difficult interpretation of deep generative models for neural data.
method Adapted variational auto-encoder to integrate task variables.
result Improves interpretability and identifiability of neural codes.
Additive decoders tackle latent variables and image generation.
problem Latent variables identification and out-of-support image generation in representation learning.
method Additive decoders that can identify latent variables up to permutation and block-wise invertible transformations, and generate novel images by recombining observed factors.
result Additive decoders provide a new setting for nonlinear independent component analysis and can generate novel images by recombining observed factors.
Variational auto-encoders (VAE) are scalable and powerful generative models. However, the choice of the variational posterior determines tractability and flexibility of the VAE. Commonly, latent variables are modeled using the normal distribution with a diagonal covariance matrix. This results in computational efficien…
The study analyzes how data augmentation helps isolate content from style in self-supervised learning.
problem Understanding how data augmentation affects the separation of content and style in self-supervised learning.
method Formulated a latent variable model with content and style components, studied identifiability of latent representation, and introduced a dataset to test the theory.
result Sufficient conditions for identifying the invariant content partition in self-supervised learning.
Many recent invertible neural architectures are based on coupling block designs where variables are divided in two subsets which serve as inputs of an easily invertible (usually affine) triangular transformation. While such a transformation is invertible, its Jacobian is very sparse and thus may lack expressiveness. Th…
New method for efficient marginalization of discrete latent variables in neural networks.
problem Computational challenges in training models with discrete latent variables.
method Parameterizing discrete distributions using sparse mappings (sparsemax and structured variants) to reduce support and enable efficient marginalization.
result Achieved good performance in various tasks with efficient and practical training.
AIKAE enhances IKAE for long-term time series forecasting.
problem Limitation of dimension conservation in IKAE models.
method Augmented with a non-invertible encoder network.
result AIKAE improves long-term forecasting accuracy.
Generative Adversarial Networks (GANs) play an increasingly important role in machine learning. However, there is one fundamental issue hindering their practical applications: the absence of capability for encoding real-world samples. The conventional way of addressing this issue is to learn an encoder for GAN via Vari…
The paper discovers a hidden component in data using an autoencoder with a discriminator.
problem Discovering a single independent latent variable in data.
method An autoencoder with a discriminator is used to recover the hidden component.
result The approach can recover the hidden component up to entropy-preserving transformations.
Develops a new method for nonlinear dimension reduction using random features.
problem Statistical challenges in generalizing Gaussian process-based latent variable models to non-Gaussian data.
method Random feature latent variable models (RFLVMs) that approximate nonlinear relationships with linear functions of random features.
result RFLVMs produce comparable results to state-of-the-art methods on various data types.
Improved DSSMs for easier interpretable latent variables.
problem Complex and hard-to-interpret latent variables in DSSMs.
method Simplified predictive decoder and shrinkage priors.
result Interpretable latent variables improve forecasting performance.
A new model decouples global and local image representations without supervision.
problem Learning decoupled global and local image representations without supervision.
method Variational auto-encoding framework with invertible generative flow.
result The model effectively learns decoupled representations of images.
Bayesian neural networks improve uncertainty quantification in non-linear dimensionality reduction.
problem Current neural network models lack adequate uncertainty quantification.
method Deploy Markov chain Monte Carlo sampling algorithms for Bayesian inference in ANN models with latent variables.
result New research directions are needed due to fundamental challenges in neural networks with latent variables.
A new method maps high-dimensional Bayesian inverse problems to lower dimensions.
problem High-dimensional Bayesian inverse problems with complex prior information.
method Data-driven VAE prior and KRnet map for posterior approximation in latent space.
result Efficiently reduces computational cost and approximates posterior distributions.
This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representati…
Deep equilibrium models estimate latent variables from data.
problem Estimating latent variables from data.
method Generalized exponential family models, deep equilibrium networks.
result Deep equilibrium models solve MAP estimates for latent and transformation parameters.
Proposes a non-parametric method for deep discrete latent variable models.
problem Learning sparse discrete latent representations in deep models.
method Iterative algorithm with Beta-Bernoulli process prior and local data scaling.
result Improves sparsity and scalability of deep discrete latent variable models.
Enhances deep kernel learning with stochastic latent variables for better model regularization.
problem Weak model regularization in deep kernel learning, especially on small datasets.
method Introduces DLVKL model with stochastic latent variables, NSDE for expressive posterior, and hybrid prior.
result DLVKL-NSDE outperforms existing deep GPs on large datasets.