Improved multimodal variational models capture more complex joint distributions.
problem Limited expressiveness of multimodal variational models.
method Used normalizing flows to approximate and transform a simple parametric joint posterior into a more complex one.
result The model improves on state-of-the-art multimodal variational methods on various tasks.
Stacking improves inference for multimodal Bayesian posterior distributions.
problem Difficulty of MCMC in moving between modes and underestimation of posterior uncertainty.
method Parallel runs of MCMC, variational, or mode-based inference, combined using Bayesian stacking.
result Stacking efficiently samples from multimodal posterior distributions and represents uncertainty better than variational inference.
BSVGD improves sampling for multimodal distributions using branching.
problem Sampling from multimodal distributions.
method Random branching in Stein Variational Gradient Descent (SVGD).
result Theoretical convergence guarantee and empirical validation.
Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
MHVAE learns cross-modality inference inspired by human cognition.
problem Cross-modality inference in multimodal data.
method Hierarchical multimodal generative model with modality-specific and joint-modality distributions.
result MHVAE performs on par with state-of-the-art models on multimodal datasets.
VERA-V uses variational inference to discover vulnerabilities in multimodal vision-language models.
problem Existing methods for jailbreaking vision-language models are brittle, limited, and focus on single attacks.
method VERA-V recasts jailbreak discovery as learning a joint posterior distribution over text-image prompts, using variational inference and three complementary strategies.
result VERA-V consistently outperforms state-of-the-art baselines, achieving up to 53.75% higher attack success rate.
PIMA autoencoders discover shared features in multimodal scientific data.
problem Discovering shared information in high-throughput scientific datasets.
method Physics-informed multimodal autoencoders (PIMA) with Gaussian mixture prior and product of experts formulation.
result Accurate cross-modal inference between images and mechanical stress-strain response in lattice metamaterials.
New method disentangles shared and private latent factors in multimodal data.
problem Challenges in disentangling shared and private latent factors in multimodal data.
method Proposes a modification to existing multimodal Variational Autoencoders (MMVAE) to better handle modality-specific variation.
result Demonstrates improved robustness of modified MMVAE to modality-specific variation.
New model improves multimodal autoencoders by learning joint and conditional distributions.
problem Limitations in recent multimodal autoencoders restrict their quality on complex datasets.
method Proposes a multistage training process with variational inference and Normalizing Flows, leveraging shared modality information.
result Achieves state-of-the-art results on benchmark datasets.
This paper introduces a spline-based method for nonparametric ADVI that handles complex posterior distributions.
problem Learning complex posterior distributions with skewness, multimodality, and bounded support.
method Develops a spline-based nonparametric approximation approach for ADVI.
result Establishes the asymptotic consistency of the derived lower bound for importance weighted autoencoder.
This research improves multimodal systems by adding a second objective and regularisation methods.
problem Improving performance of multimodal systems with multiple objectives and regularisation.
method Introduces a second objective over multimodal fusion using variational inference and regularisation methods.
result Demonstrates potential for multiple objectives and probabilistic methods to lower variance and improve generalisation.
Enhances multimodal generation with Normalizing Flows and correlation analysis.
problem Generating coherent cross-modal data from multiple sources.
method Uses Deep Canonical Correlation Analysis for shared information, Normalizing Flows for diversity, and Product of Experts for scalability.
result Improves likelihood, diversity, and coherence in conditional generation.
MC Dropout is re-evaluated as not Bayesian, affecting predictive posterior and multimodality.
problem MC Dropout's Bayesian properties and predictive posterior are questioned.
method Re-evaluation of MC Dropout's properties, including a new VI engine in pytorch.
result MC Dropout does not produce a faithful Bayesian predictive posterior.
Integrating deep learning with latent state space models has the potential to yield temporal models that are powerful, yet tractable and interpretable. Unfortunately, current models are not designed to handle missing data or multiple data modalities, which are both prevalent in real-world data. In this work, we introdu…
Improved phylogenetic inference using VBPI-Mixtures for tree topology and branch length.
problem Multimodality of tree-topology posterior distributions in phylogenetic inference.
method VBPI-Mixtures algorithm that uses mixture learning within the BBVI framework.
result VBPI-Mixtures captures tree-topology distributions better than VBPI.
Variational inference struggles with weight symmetries in neural networks, leading to biased posteriors.
problem Weight space symmetries in neural networks cause multimodal posteriors, challenging variational inference.
method Developed a symmetrization mechanism to create permutation invariant variational posteriors.
result Symmetrized variational posteriors have a better fit to the true posterior and improved predictive performance.
Paper bridges VI and GFlowNets, showing their equivalence in certain cases.
problem Modeling distributions over continuous and discrete structures.
method Demonstrates equivalence between VI and GFlowNets in specific scenarios.
result GFlowNets are more suitable for off-policy training without high gradient variance.
New method learns complex, multimodal distributions in ADVI.
problem Learning unimodal approximate posteriors limits ADVI's ability to capture complex data structures.
method Use stratified sampling to allow mixture distributions as approximate posteriors, derive a tighter evidence lower bound.
result SIWAE objective allows ADVI to learn more complex, multimodal distributions, improving accuracy and calibration.
One of the core problems in variational inference is a choice of approximate posterior distribution. It is crucial to trade-off between efficient inference with simple families as mean-field models and accuracy of inference. We propose a variant of a greedy approximation of the posterior distribution with tractable bas…
FTIP uses normalizing flows to improve posterior inference in function space.
problem Challenges in posterior inference with implicit-process priors.
method FTIP uses normalizing flows to define a richer variational distribution over combination weights.
result FTIP captures asymmetric and multimodal posterior structure better than Gaussian coefficient approximations.
EigenVI uses orthogonal function expansions for efficient variational inference.
problem Efficiently approximate complex distributions in variational inference.
method EigenVI constructs variational approximations using orthogonal function expansions, minimizing Fisher divergence.
result EigenVI provides more accurate approximations than existing methods for Gaussian BBVI.
Deep neural networks (DNNs) provide state-of-the-art results for a multitude of applications, but the approaches using DNNs for multimodal audiovisual applications do not consider predictive uncertainty associated with individual modalities. Bayesian deep learning methods provide principled confidence and quantify pred…
In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative f…
Stein variational gradient descent improves inference in Gaussian process models.
problem Inference in Gaussian process models with non-Gaussian likelihoods and large data volumes is computationally intensive and inaccurate with traditional methods.
method Stein variational gradient descent (SVGD) for non-parametric inference.
result SVGD monotonically decreases the Kullback-Leibler divergence from the sampling distribution to the true posterior.
An infinite parallel tempering bouncy particle sampler improves sampling efficiency for multimodal distributions.
problem Sampling from complex posterior distributions with high accuracy and efficiency.
method Introduced an infinite parallel tempering bouncy particle sampler (BPS-PT) to accelerate convergence.
result Demonstrated improved sampling efficiency for multimodal distributions through numerical simulations.
Multimodal fusion is considered a key step in multimodal tasks such as sentiment analysis, emotion detection, question answering, and others. Most of the recent work on multimodal fusion does not guarantee the fidelity of the multimodal representation with respect to the unimodal representations. In this paper, we prop…
Multimodal sensory data resembles the form of information perceived by humans for learning, and are easy to obtain in large quantities. Compared to unimodal data, synchronization of concepts between modalities in such data provides supervision for disentangling the underlying explanatory factors of each modality. Previ…
The paper develops efficient algorithms for variational inference with mixtures of isotropic Gaussians.
problem Efficiently approximating multimodal Bayesian posteriors.
method Develops a variational framework and efficient algorithms for mixtures of isotropic Gaussians.
result The approach provides accurate approximations of multimodal Bayesian posteriors while being memory and computationally efficient.
Develops a method for learning proposals in nested importance samplers.
problem Improving sampling quality in complex distributions.
method Nested Variational Inference (NVI) using forward or reverse KL divergence.
result Optimizing nested objectives leads to improved sample quality.
Mathematical analysis shows annealing prevents mode collapse in Gaussian mixtures.
problem Mode collapse in variational inference for multimodal distributions.
method Analyzed annealing strategies for Gaussian mixtures, derived formulas, and tested on neural networks.
result Appropriately chosen annealing schemes can robustly prevent mode collapse.
VAEs struggle with surjective multimodal data, especially class labels describing images.
problem VAEs struggle to capture variability in surjective multimodal data.
method Theoretical and empirical demonstration of VAEs with a mixture of experts posterior.
result VAEs with a mixture of experts posterior can disregard variation in surjective multimodal data.
DMSTF models spatio-temporal data with deep Markov priors.
problem Analyzing nonlinear multimodal spatio-temporal dynamics.
method Deep Markov spatio-temporal factorization with stochastic variational inference.
result DMSTF outperforms other methods in predictive performance and clustering.
NVA combines variational posteriors, annealing, and natural-gradient learning for multimodal optimization.
problem Finding multiple global and local modes in nonconvex objectives.
method NVA integrates variational posteriors, annealing, and natural-gradient learning.
result NVA outperforms gradient descent and evolution strategies on simulations and real-world problems.
Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations. In this paper, we extend their application to estimate \emph{structured} approximations such as mixtures of E…
We introduce a Bayesian framework for inference with a supervised version of the Gaussian process latent variable model. The framework overcomes the high correlations between latent variables and hyperparameters by using an unbiased pseudo estimate for the marginal likelihood that approximately integrates over the late…
This paper describes InfoCatVAE, an extension of the variational autoencoder that enables unsupervised disentangled representation learning. InfoCatVAE uses multimodal distributions for the prior and the inference network and then maximizes the evidence lower bound objective (ELBO). We connect the new ELBO derived for …
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
A new method improves posterior approximation for complex distributions.
problem Difficulty in capturing multimodal and heavy-tailed posteriors with standard normalizing flows.
method StiCTAF: stick-breaking mixture base with component-wise tail adaptation.
result Improved tail recovery and better mode coverage compared to benchmarks.
In the original version of the Variational Autoencoder, Kingma et al. assume Gaussian distributions for the approximate posterior during the inference and for the output during the generative process. This assumptions are good for computational reasons, e.g. we can easily optimize the parameters of a neural network usi…
Proposes a differentiable LSE-ICNN for modeling multi-well potentials.
problem Modeling multi-well potentials in various scientific domains.
method Log-sum-exponential (LSE) mixture of input convex neural network (ICNN) modes.
result Smooth surrogate that retains convexity within basins and allows gradient-based learning.
AECF improves multimodal inference robustness and calibration.
problem Robustness and calibration issues in multimodal systems with missing inputs.
method Adaptive Entropy-Gated Contrastive Fusion (AECF) layer.
result Improves masked-input mAP by +18 pp at a 50% drop rate.
Variational inference (VI) provides fast approximations of a Bayesian posterior in part because it formulates posterior approximation as an optimization problem: to find the closest distribution to the exact posterior over some family of distributions. For practical reasons, the family of distributions in VI is usually…
Method for initializing Gaussian mixtures for variational inference with multi-modal distributions.
problem Challenges in variational inference with Gaussian mixtures due to multimodality and nonconvex loss functions.
method Optimization to find local maxima, local Gaussian approximations, and constrained least squares regression.
result Robust initialization improves variational inference performance and scalability.
Learning a generative model from partial data (data with missingness) is a challenging area of machine learning research. We study a specific implementation of the Auto-Encoding Variational Bayes (AEVB) algorithm, named in this paper as a Variational Auto-Decoder (VAD). VAD is a generic framework which uses Variational…
A new method improves Bayesian inference for multimodal posteriors.
problem Insensitivity to well-separated modes in multimodal posteriors.
method Weighted Kernel Stein Discrepancy method.
result Significantly improved mode sensitivity compared to standard KSD-Bayes.
Integrates MRF into multimodal VAE for better complex intermodal interactions.
problem Lack of effective modeling of complex intermodal interactions in multimodal VAEs.
method Incorporates Markov Random Field into prior and posterior distributions of multimodal VAE.
result Demonstrates superior performance in managing complex intermodal dependencies.
FACTM combines FA with correlated topic modeling for structured data integration.
problem Integrating structured data modalities like text and single cell sequencing.
method Bayesian FACTM model combining FA and correlated topic modeling with variational inference.
result FACTM outperforms other methods in identifying clusters in structured data and integrating them with simple modalities.
Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing …