Bayesian deep learning faces posterior collapse due to likelihood vs. prior competition.
problem Posterior collapse in Bayesian deep learning models.
method Identified competition between likelihood and prior regularization in a linear latent variable model.
result Posterior collapse is related to neural and dimensional collapse, suggesting a broader learning issue.
Deep Gaussian Processes with polynomial kernels can collapse rapidly without proper hyperparameter tuning.
problem The collapse of Deep Gaussian Processes with polynomial kernels without careful hyperparameter tuning.
method Analysis using the Berry-Esseen Theorem and observation of prior behavior.
result The prior of a Deep Gaussian Process collapses rapidly towards zero or places negligible mass on low norm functions without proper hyperparameter tuning.
High-dimensional VAEs inevitably collapse to prior, requiring large datasets for good performance.
problem Posterior collapse in VAEs leads to poor representation learning quality.
method Analyzed a minimal VAE in a high-dimensional limit, evaluating conditions for posterior collapse with respect to beta and dataset size.
result VAEs face 'inevitable posterior collapse' beyond a certain beta threshold, regardless of dataset size.
End-to-end learnable Gaussian mixture priors improve diffusion models' exploration and expressiveness.
problem Challenges in diffusion models when priors differ from target distributions.
method End-to-end learnable Gaussian mixture priors (GMPs) with iterative refinement.
result Significant performance improvements across various benchmark problems.
MMCGAN uses explicit manifold learning to improve GAN performance.
problem GAN mode collapse and unstable training.
method Introduces Minimum Manifold Coding (MMC) as a prior to guide GAN training.
result MMCGAN effectively alleviates mode collapse and stabilizes GAN training.
New method controls posterior collapse in VAEs without network architecture constraints.
problem Posterior collapse in VAEs reduces diversity of generated samples.
method Introduces Latent Reconstruction (LR) loss to control posterior collapse.
result Controls posterior collapse on various datasets without architectural constraints.
New method controls posterior collapse in VAEs with theoretical guarantees.
problem Posterior collapse in VAEs where encoder ignores latent structure.
method Inverse Lipschitz constraint on decoder network.
result Controls degree of posterior collapse for various VAE models.
New metrics prevent event collapse in contrast maximization frameworks.
problem Event collapse in contrast maximization frameworks.
method First principles of space-time deformation based on differential geometry and physics.
result Proposed metrics mitigate event collapse and do not harm well-posed warps.
JORC-UMAP improves UMAP by incorporating geometric and topological priors.
problem UMAP's local Euclidean distance assumption fails to capture intrinsic manifold geometry, leading to topological tearing and structural collapse.
method JORC-UMAP introduces Ollivier-Ricci curvature as a geometric prior and Jaccard similarity as a topological prior to reinforce edges and reduce redundant links.
result JORC-UMAP reduces tearing and collapse more effectively than standard UMAP and other DR methods, as measured by SVM accuracy and triplet preservation scores.
New critics improve VAEs by preventing latent collapse.
problem Posterior collapse in VAEs where latent variables are ignored.
method Inference critics that detect and incentivize meaningful latent representations.
result Optimizing inference critics increases mutual information between latent and observations.
A new meta-learning method using shared variational inference.
problem Meta-learning with uncertainty over model parameters.
method Shared amortized variational inference network for conditional prior and posterior.
result Prevents collapse of conditional prior to Dirac delta function.
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Prob…
This paper proposes Dirichlet Variational Autoencoder (DirVAE) using a Dirichlet prior for a continuous latent variable that exhibits the characteristic of the categorical probabilities. To infer the parameters of DirVAE, we utilize the stochastic gradient method by approximating the Gamma distribution, which is a comp…
Bayesian models that mix multiple Dirichlet prior parameters, called Multi-Dirichlet priors (MD) in this paper, are gaining popularity. Inferring mixing weights and parameters of mixed prior distributions seems tricky, as sums over Dirichlet parameters complicate the joint distribution of model parameters. This paper s…
NE-VAE prevents posterior collapse in VAEs by embedding neighbors in latent space.
problem Posterior collapse in VAEs when strong decoders are used.
method Neighbor embedding in latent space to prevent collapse.
result NE-VAE produces qualitatively different latent representations with active latent dimensions.
A fast geometric regularizer improves event camera performance.
problem Event collapse in contrast maximization framework.
method Geometric regularizer to mitigate overfitting.
result State-of-the-art accuracy with reduced computational complexity.
This work tackles posterior collapse in conditional and hierarchical VAEs.
problem Posterior collapse in VAEs leads to poor latent variable representations.
method Theoretical analysis of linear conditional and hierarchical VAEs, empirical validation.
result Theoretical and empirical evidence of posterior collapse causes in conditional and hierarchical VAEs.
Variational autoencoders often collapse, showing latent variables are non-identifiable.
problem Posterior collapse in variational autoencoders due to non-identifiable latent variables.
method Proves latent variable non-identifiability causes posterior collapse. Proposes latent-identifiable models using Brenier maps and input convex neural networks.
result Latent-identifiable models resolve posterior collapse and provide meaningful representations.
Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," …
In narrow asymptotic settings Gaussian VAE models of continuous data have been shown to possess global optima aligned with ground-truth distributions. Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice. However, contrary to c…
KL annealing helps VAEs avoid posterior collapse and overfitting.
problem Posterior collapse and overfitting in VAEs.
method Theoretical analysis of learning dynamics with KL annealing.
result Posterior collapse is inevitable when β exceeds a threshold. A new model tackles language generation issues by using discrete variational attention.
problem Information under-representation and posterior collapse in variational autoencoders.
method Proposes a discrete variational attention model with categorical distribution over attention mechanism.
result Enhances latent space for language generation and avoids posterior collapse.
We address feature interpretation and reproducibility issues in dense nets, proposing a modified loss function.
problem Feature interpretation and reproducibility issues in dense nets.
method Proposed a modified loss function to circumvent basis collapse.
result Substantially concise nets with 100x fewer parameters and lower MSE loss.
COME replaces entropy minimization to prevent model collapse.
problem Overconfidence in entropy minimization leads to model collapse.
method COME explicitly models uncertainty with a Dirichlet prior distribution.
result COME achieves state-of-the-art performance on various TTA settings.
We introduce a Bayesian Gaussian process latent variable model that explicitly captures spatial correlations in data using a parameterized spatial kernel and leveraging structure-exploiting algebra on the model covariance matrices for computational tractability. Inference is made tractable through a collapsed variation…
New data accumulation prevents model collapse in generative models.
problem Model collapse in generative models trained on their own outputs.
method Empirical study of language models, diffusion models, and variational autoencoders; analytically tractable framework for linear models.
result Accumulating synthetic data alongside real data avoids model collapse, preventing performance degradation.
Molecule generation is to design new molecules with specific chemical properties and further to optimize the desired chemical properties. Following previous work, we encode molecules into continuous vectors in the latent space and then decode the vectors into molecules under the variational autoencoder (VAE) framework.…
The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be opti…
We introduce a novel approach for estimating Latent Dirichlet Allocation (LDA) parameters from collapsed Gibbs samples (CGS), by leveraging the full conditional distributions over the latent variable assignments to efficiently average over multiple samples, for little more computational cost than drawing a single addit…
Generative models produce realistic objects in many domains, including text, image, video, and audio synthesis. Most popular models---Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs)---usually employ a standard Gaussian distribution as a prior. Previous works show that the richer family of pri…
Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poore…
Study rigidifies torus bundles under first Betti number constraints.
problem Understanding the structure of torus fibrations under first Betti number restrictions.
method Established rigidity results and necessary/sufficient conditions for topological splitting.
result Classification of torus bundles under specific Betti number constraints.
Framework expands particle filtering to estimate states beyond prior boundaries.
problem Limitations of traditional particle filtering in estimating states outside prior support.
method Diffusion-Enhanced Particle Filtering Framework with adaptive diffusion, entropy-driven regularisation, and kernel-based perturbations.
result Framework significantly improves state estimation accuracy and success rates for out-of-boundary targets.
Improved co-clustering for robust data analysis.
problem Noise and posterior collapse in co-clustering.
method Fully variational Co-clustering framework with Compositional ELBO.
result Enhanced robustness and accuracy in co-clustering.
In this paper we focus on preprocessing for persistent homology computations. We adapt some techniques which were successfully used for standard homology computations. The main idea is to reduce the complex prior to generating its boundary matrix, which is costly to store and process. We discuss the following reduction…
VINNAS uses variational inference to avoid mode collapse in neural architecture search.
problem Mode collapse in gradient-based NAS methods, leading to suboptimal architectures.
method Differentiable variational inference with variational dropout and automatic relevance determination.
result State-of-the-art accuracy with up to twice fewer non-zero parameters.
Recent advances in weakly supervised classification allow us to train a classifier only from positive and unlabeled (PU) data. However, existing PU classification methods typically require an accurate estimate of the class-prior probability, which is a critical bottleneck particularly for high-dimensional data. This pr…
Blade uses diffusion priors to accurately and calibratedly infer complex systems.
problem Derivative-free Bayesian inversion for high-dimensional, nonlinear problems with costly forward models.
method Blade employs an ensemble of interacting particles and diffusion models as priors, querying forward models only through evaluations.
result Blade produces well-calibrated posterior samples that existing methods cannot, improving with more iterations and particles.
A method for eliciting expert beliefs using preferential questions and normalizing flows.
problem Eliciting high-dimensional probability distributions from noisy judgments.
method Normalizing flows based on preferential questions with a novel functional prior.
result The method allows for the inference of arbitrarily flexible densities from preferential judgments.
We exhibit a concentration-collapse decomposition of singularities of fourth order curvature flows, including the L2 curvature flow and Calabi flow, in dimensions n≤4. The proof requires the development of several new a priori estimates. First, we develop a smoothing result for initial metrics with small ener…
Study uses satellite data to predict tailings dam collapse risk.
problem Detecting early signs of tailings dam instability.
method Spectral analysis of satellite InSAR displacement time series data.
result Algorithm detects risk milestones up to 5 months before dam collapse.
Due to the phenomenon of "posterior collapse," current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires augmenting the objective so it does not only maximize the likelihood of the data. In this paper, we propose an alternative that utilizes t…
The edge partition model (EPM) is a fundamental Bayesian nonparametric model for extracting an overlapping structure from binary matrix. The EPM adopts a gamma process (ΓP) prior to automatically shrink the number of active atoms. However, we empirically found that the model shrinkage of the EPM does not typically wo…
A new framework CyGen models joint distributions using cyclic conditionals.
problem Modeling a joint distribution using only two conditional models without relying on an uninformative prior.
method Developed a general theory for operable equivalence criteria for compatibility and sufficient conditions for determinacy. Proposed CyGen framework and methods to achieve compatibility and determinacy.
result CyGen better fits data and captures more representative features compared to models using an uninformative prior.
Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes a uni-modal standard Gaussian prior on the latent variables. This indu…
A new model predicts discrete events with flexible, nonparametric baseline and excitation.
problem Limited flexibility in discrete Hawkes models for event prediction.
method Gaussian Process Discrete Hawkes Process (GP-DHP) with collapsed latent representation.
result Improves predictive log-likelihood for diverse event patterns.
New model detects hidden group structures in criminal networks.
problem Challenges in identifying group structures in criminal networks with noisy data.
method Developed an extended stochastic block model (ESBM) to infer group structures.
result Unveiled complex block structures in an Italian mafia network.
Polynomial-time algorithm learns ReLU networks without assumptions.
problem Learning linear combinations of ReLU activations with Gaussian inputs.
method Random contractions of moment tensors and multi-scale analysis.
result First polynomial-time algorithm without additional assumptions.