Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of j…
Develops a contrastive framework for data-efficient multimodal learning.
problem Expensive training of multimodal generative models requiring related multimodal data.
method Contrastive framework for multimodal learning, distinguishing related from unrelated data.
result Data-efficient multimodal learning on challenging datasets for various VAE models.
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) mode…
Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exp…
Class-conditional generative models are crucial tools for data generation from user-specified class labels. Existing approaches for class-conditional generative models require nontrivial modifications of backbone generative architectures to model conditional information fed into the model. This paper introduces a plug-…
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
Study analyzes bias interactions in multimodal models using simulation-based methods.
problem Analyzing dynamic bias interactions in multimodal models to ensure fairness and equity.
method Simulation-based heuristic approach to compute bias scores for text-only, image-only, and multimodal embeddings.
result Multimodal bias interactions can be amplification, mitigation, or neutral, with text bias often dominant.
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised mult…
The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental rese…
Bayesian VFLMSP improves multimodal survival prediction with privacy.
problem Privacy and reliability in multimodal time-to-event prediction.
method Bayesian Vertical Federated Learning (VFL) with differential privacy.
result Consistent improvements in C-index compared to existing methods.
Improved software flaw detection using NAS on multimodal DL models.
problem Software flaw detection in multimodal deep learning models.
method Adapted NAS framework for multimodal learning, combined with multimodal deep learning models.
result Improved performance on the Juliet Test Suite.
With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words re…
In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative f…
Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent r…
Proposes a new method for analyzing multimodal neuroimaging data.
problem Combining interpretability and flexibility in multimodal data analysis.
method Orthogonalized kernel debiased machine learning approach.
result Established consistency and asymptotic normality of the estimated primary parameter.
Study quantifies interactions between unlabeled multimodal data.
problem Understanding how modalities combine in semi-supervised settings.
method Information-theoretic definitions and bounds derivation.
result Validated lower and upper bounds accurately track true interactions.
DECAT framework evaluates multimodal models for shared biology, detecting confounders and false positives.
problem Determining if multimodal models learn shared biology or just confounders.
method DECAT framework classifies multimodal representations into four diagnostic scenarios using null-referenced metrics.
result DECAT detects confounders and false positives in multimodal models, improving with larger cohorts and stronger representations.
Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs)…
Improved multimodal variational models capture more complex joint distributions.
problem Limited expressiveness of multimodal variational models.
method Used normalizing flows to approximate and transform a simple parametric joint posterior into a more complex one.
result The model improves on state-of-the-art multimodal variational methods on various tasks.
Improved text-to-image and multimodal understanding through adaptive generation order optimization.
problem Determining optimal generation sequences in text-to-image synthesis and multimodal understanding.
method Introduced a learnable control module trained via Group Relative Policy Optimization (GRPO) to determine the generation order.
result Learning the control block substantially improves text-to-image alignment and multimodal understanding in DLMs.
Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for class…
Multimodal deep learning improves flaw detection in software programs.
problem Current flaw detection relies on single software representations.
method Adapted multimodal deep learning models for flaw detection.
result Multimodal models outperform traditional deep learning models.
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
New ELBO formulation improves multimodal learning.
problem Existing ELBO models struggle with multimodal data.
method Proposes a generalized ELBO for multimodal data.
result Demonstrates improved performance over state-of-the-art models.
DFMs enable flow-based models for multimodal discrete and continuous data.
problem Combining discrete and continuous data for generative models.
method Discrete Flow Models (DFMs) using Continuous Time Markov Chains.
result DFMs achieve state-of-the-art co-design performance for protein structure and sequence generation.
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.
Unified approach for multimodal data prediction using synthetic data generation.
problem Challenges in integrating heterogeneous data types for accurate predictive performance.
method Generative Distribution Prediction (GDP) framework that uses multimodal synthetic data generation.
result Empirical validation across four tasks demonstrates versatility and effectiveness of GDP.
Hybrid model for multimodal distributions using diffusion and classification.
problem Sampling from multimodal distributions with correct proportions.
method Divide-and-conquer strategy: identify modes, train classifiers, diffusion models, bridge sampling.
result Framework effectively handles multimodal distributions in high dimensions.
Learning social media data embedding by deep models has attracted extensive research interest as well as boomed a lot of applications, such as link prediction, classification, and cross-modal search. However, for social images which contain both link information and multimodal contents (e.g., text description, and visu…
Characterizing the dynamic interactive patterns of complex systems helps gain in-depth understanding of how components interrelate with each other while performing certain functions as a whole. In this study, we present a novel multimodal data fusion approach to construct a complex network, which models the interaction…
Open-FinLLMs tackle financial tasks with multimodal capabilities.
problem Financial LLMs lack multimodal capabilities and real-world applicability.
method Developed Open-FinLLMs, an open-source multimodal financial LLM suite.
result Open-FinLLMs outperform advanced financial and general LLMs in diverse tasks.
New model learns multimodal data better than DAGs.
problem Complex multimodal data not well captured by DAGs.
method Latent partial causal model with two latent coupled variables.
result Identifiability result shows representations correspond to latent variables.
Systematic review of multimodal data challenges and solutions.
problem Challenges in integrating diverse data types for improved diagnostics and personalized care.
method Synthesizing findings from 69 studies on technical obstacles and recent methodological advances.
result Promising solutions like transfer learning, generative models, attention mechanisms, and neural architecture search.
This paper strengthens the computational separation between multimodal and unimodal learning, showing unimodal learning is hard on typical instances.
problem Theoretical justification for empirical success of multimodal machine learning.
method Introduced a stronger average-case computational separation between unimodal and multimodal learning.
result For typical instances, unimodal learning is computationally hard, while multimodal learning is easy.
A new objective function using Jensen-Shannon divergence improves generative learning from multiple data types.
problem Learning from multiple data types efficiently and accurately.
method Proposes a novel objective function using Jensen-Shannon divergence to approximate multimodal posteriors directly.
result The mmJSD objective optimizes an ELBO and improves generative learning tasks.
lamBERT learns language and actions using multimodal BERT.
problem Learning language and actions in complex environments.
method Extending BERT to multimodal representation and integrating with reinforcement learning.
result lamBERT model achieved higher rewards in multitask and transfer settings.
There has been an increased interest in multimodal language processing including multimodal dialog, question answering, sentiment analysis, and speech recognition. However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption. To address these c…
Evidential Softmax preserves multimodality in sparse probability distributions for generative models.
problem Sparse probability distributions in deep generative models make exact marginalization computationally intractable.
method Introduce ev-softmax, a sparse normalization function that preserves multimodality and can be trained with probabilistic loss functions.
result ev-softmax outperforms existing techniques in distributional accuracy and dimensionality reduction.
MHVAE learns cross-modality inference inspired by human cognition.
problem Cross-modality inference in multimodal data.
method Hierarchical multimodal generative model with modality-specific and joint-modality distributions.
result MHVAE performs on par with state-of-the-art models on multimodal datasets.
Estimates interactions between modalities for multimodal data.
problem Accurately quantifying interactions between different data types.
method Developed Lightweight Sample-wise Multimodal Interaction (LSMI) estimator using pointwise information theory.
result LSMI reveals fine-grained dynamics in multimodal data.
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
MDNs offer a data-efficient alternative to diffusion and flow models for multimodal scientific learning.
problem Capturing multimodal conditional uncertainty in scientific inverse problems.
method Mixture Density Networks (MDNs) as explicit parametric density estimators.
result MDNs achieve superior generalization, interpretability, and sample efficiency in scientific tasks.
Multimodal fusion is considered a key step in multimodal tasks such as sentiment analysis, emotion detection, question answering, and others. Most of the recent work on multimodal fusion does not guarantee the fidelity of the multimodal representation with respect to the unimodal representations. In this paper, we prop…
Unified framework explains few-shot multimodal medical imaging performance.
problem Limited labeled data in rare diseases and low-resource settings.
method PAC learning, VC theory, PAC Bayesian analysis, information gain, Chain of Thought reasoning.
result Unified theoretical framework for few-shot multimodal medical imaging.
Proposes a method to improve SLMC for multimodal distributions.
problem Difficulty of applying SLMC to multimodal distributions.
method Parallel adaptive annealing with VAE-SLMC.
result Can proficiently obtain accurate samples from multimodal distributions.
New framework estimates graph from multimodal functional data.
problem Estimating graph from joint multimodal functional data.
method Integrative framework using partial correlation operator.
result Estimator converges to stationary point with quantifiable error.
Graph-based multimodal federated learning for HAR improves accuracy and privacy.
problem Challenges in HAR due to noisy data, incomplete measurements, and privacy concerns.
method Proposes GraMFedDHAR, a Graph-based Multimodal Federated Learning framework for HAR tasks, using modality-specific graphs, residual GCNs, and attention-based fusion.
result Experimental results show up to 13 percent performance improvement for MultiModalGCN under differential privacy constraints.