Plug-and-play multimodal controller improves class-conditional image generation.
problem Generating class-conditional images from user-specified labels.
method Introduces a `multimodal controller` to generate multimodal data without additional learning parameters.
result Multimodal controlled generative models produce higher quality class-conditional images and novel modalities.
Improved text-to-image and multimodal understanding through adaptive generation order optimization.
problem Determining optimal generation sequences in text-to-image synthesis and multimodal understanding.
method Introduced a learnable control module trained via Group Relative Policy Optimization (GRPO) to determine the generation order.
result Learning the control block substantially improves text-to-image alignment and multimodal understanding in DLMs.
A new GAN model LDAGAN uses Latent Dirichlet Allocation to model multimodal images.
problem Ignoring the structure and multimodal characteristics of vision data in GANs.
method Introduced a Dirichlet prior for multimodal image generation leading to LDAGAN. LDAGAN defines generative modes for each sample and uses a VEM algorithm for adversarial training.
result Experimental results show LDAGAN outperforms other GANs on real-world datasets.
The paper introduces multimodal generative models to improve data marginal likelihood.
problem Improving data marginal likelihood in multimodal settings.
method Derives variational bounds on the evidence for multimodal deep generative models, generalizes objectives for different model types, and benchmarks across various datasets.
result Multimodal VAEs excel in image, label, and text datasets with and without weak supervision.
Multimodal bitransformer boosts image-text classification.
problem Combining text and image modalities for improved classification.
method Supervised multimodal bitransformer model integrating text and image encoders.
result State-of-the-art performance on multimodal classification benchmarks.
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
InVA models image outcomes from multiple modalities, outperforming standard VAEs.
problem Understanding relationships across multiple imaging modalities in neuroimaging.
method Integrative Variational Autoencoder (InVA) framework for image-on-image regression.
result InVA accurately predicts PET scans from structural MRI, outperforming conventional models.
VAEs struggle with surjective multimodal data, especially class labels describing images.
problem VAEs struggle to capture variability in surjective multimodal data.
method Theoretical and empirical demonstration of VAEs with a mixture of experts posterior.
result VAEs with a mixture of experts posterior can disregard variation in surjective multimodal data.
Learning social media data embedding by deep models has attracted extensive research interest as well as boomed a lot of applications, such as link prediction, classification, and cross-modal search. However, for social images which contain both link information and multimodal contents (e.g., text description, and visu…
Proposes neural network for causal inference with multimodal data.
problem Estimating causal effects with text and image data as confounders.
method Double machine learning framework adapted to partially linear models, semi-synthetic dataset generation.
result Improved performance in causal effect estimation with multimodal data.
StoryGen uses images to generate coherent text from multiple images.
problem Generating coherent text from multiple images.
method Designing a Relational Text Data Generator algorithm to relate features from plural images.
result The model can generate meaningful paragraphs containing extracted features from input images.
New results show contrastive learning can recover shared factors in multimodal data.
problem Understanding when contrastive learning can recover shared latent factors in multimodal data.
method New identifiability results for multimodal contrastive learning, distinguishing between multi-view and multimodal settings.
result Contrastive learning can block-identify shared latent factors in multimodal data, even with dependencies.
Unified framework explains few-shot multimodal medical imaging performance.
problem Limited labeled data in rare diseases and low-resource settings.
method PAC learning, VC theory, PAC Bayesian analysis, information gain, Chain of Thought reasoning.
result Unified theoretical framework for few-shot multimodal medical imaging.
Study analyzes bias interactions in multimodal models using simulation-based methods.
problem Analyzing dynamic bias interactions in multimodal models to ensure fairness and equity.
method Simulation-based heuristic approach to compute bias scores for text-only, image-only, and multimodal embeddings.
result Multimodal bias interactions can be amplification, mitigation, or neutral, with text bias often dominant.
Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches, however, primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in knowledge bases, such as text, …
Develops a deep multimodal classifier for medical images and reports.
problem Challenges of small datasets in medical deep learning.
method Transfer learning, integrated gradients, unsupervised clustering, tuning parameter.
result Improves classification accuracy by 4% and 7% on average.
Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution i…
Training-free method improves text-to-image generation quality.
problem Error accumulation in simultaneous token updates of masked diffusion models.
method Training-free self-correction framework exploiting inductive biases.
result Significantly improved generation quality on text-to-image tasks.
Proposes MR-SNE for multimodal data visualization.
problem Visualizing data from multiple domains with relations across them.
method Extends t-SNE to compute augmented relations and jointly embed them in a low-dimensional space.
result Demonstrates promising performance in visualizing Flickr and Animal with Attributes 2 datasets.
New method improves diversity in GAN-generated images.
problem Lack of diversity in GAN-generated images.
method Introducing moment reconstruction losses to replace the reconstruction loss in GAN training.
result Improved diversity in generated images without sacrificing visual fidelity.
New method tackles heterogeneous breast cancer imaging data.
problem Challenges in multimodality breast cancer imaging data.
method Developed a multilayer tensor learning method to incorporate heterogeneity.
result Outperforms other existing methods in predicting disease status.
New deep fusion methods improve human action recognition using depth and inertial sensor data.
problem Existing multimodal HAR frameworks lack mid-level feature fusion.
method Proposes three deep multilevel multimodal fusion frameworks, transforming depth and inertial sensor data into images and using convolution with Prewitt filter to create modality within modality.
result Supremacy of proposed fusion frameworks over existing methods on three publicly available datasets.
New method handles missing data in multimodal brain imaging.
problem Missing data in multimodal brain imaging.
method Full Information Linked ICA (FI-LICA) algorithm.
result FI-LICA outperforms current practices in classification and prediction.
VERA-V uses variational inference to discover vulnerabilities in multimodal vision-language models.
problem Existing methods for jailbreaking vision-language models are brittle, limited, and focus on single attacks.
method VERA-V recasts jailbreak discovery as learning a joint posterior distribution over text-image prompts, using variational inference and three complementary strategies.
result VERA-V consistently outperforms state-of-the-art baselines, achieving up to 53.75% higher attack success rate.
Dp-CLIP preserves privacy in multimodal AI training.
problem Privacy concerns in multimodal AI, especially in vision-language tasks.
method Differentially private adaptation of CLIP model.
result Dp-CLIP retains accuracy while ensuring privacy.
We tackle here the problem of multimodal image non-rigid registration, which is of prime importance in remote sensing and medical imaging. The difficulties encountered by classical registration approaches include feature design and slow optimization by gradient descent. By analyzing these methods, we note the significa…
Unified approach for multimodal data prediction using synthetic data generation.
problem Challenges in integrating heterogeneous data types for accurate predictive performance.
method Generative Distribution Prediction (GDP) framework that uses multimodal synthetic data generation.
result Empirical validation across four tasks demonstrates versatility and effectiveness of GDP.
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared …
New method uses adversarial networks to improve image quality in autoencoders.
problem Blurriness in autoencoder-generated images due to Gaussian assumptions.
method Integrates adversarial networks to optimize parameters without Gaussian assumptions.
result Improves image quality by allowing better representation of multimodal distributions.
UniFinEval benchmarks financial models across text, images, and videos.
problem Challenges in evaluating financial multimodal models across text, images, and videos.
method Proposes UniFinEval, a unified multimodal benchmark for financial scenarios.
result Gemini-3-pro-preview achieves best performance but still lags behind experts.
Method retrieves similar fashion items from images and text, enabling style refinement.
problem Lack of intuitive, interactive refinement in search engines for fashion items.
method Joint visual-textual embedding training, Mini-Batch Match Retrieval, attribute extraction.
result Improved performance in multimodal style search, demonstrated through benchmark.
This paper investigates uncertainty calibration in multimodal large language models.
problem Challenges in properly calibrating uncertainty in multimodal large language models.
method Investigation of representative MLLMs across various scenarios, including visual fine-tuning and multimodal training.
result MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments.
Improved software flaw detection using NAS on multimodal DL models.
problem Software flaw detection in multimodal deep learning models.
method Adapted NAS framework for multimodal learning, combined with multimodal deep learning models.
result Improved performance on the Juliet Test Suite.
A method to improve image synthesis diversity using mutual information.
problem Mode collapse in conditional GANs for multimodal image synthesis.
method Explicitly estimate and maximize mutual information between latent code and output image.
result Prevents mode collapse and encourages synthesis of diverse images.
Systematic review of multimodal data challenges and solutions.
problem Challenges in integrating diverse data types for improved diagnostics and personalized care.
method Synthesizing findings from 69 studies on technical obstacles and recent methodological advances.
result Promising solutions like transfer learning, generative models, attention mechanisms, and neural architecture search.
We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual analysis and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the laten…
Current high-throughput data acquisition technologies probe dynamical systems with different imaging modalities, generating massive data sets at different spatial and temporal resolutions posing challenging problems in multimodal data fusion. A case in point is the attempt to parse out the brain structures and networks…
Push-forward models struggle to fit multimodal distributions due to high Lipschitz constants.
problem Expressivity of push-forward generative models in fitting multimodal distributions.
method Analyzing the Lipschitz constant and its relation to the total variation distance and Kullback-Leibler divergence.
result Push-forward models require high Lipschitz constants to approximate multimodal distributions, leading to a trade-off between expressivity and stability.
This paper explores how the latent structure affects clustering in GAN-generated data.
problem Achieving well-clustered data in GAN-generated spaces, especially with class imbalance.
method Derives conditions for faithful clustering in GANs, including multimodal latent space, latent space inversion, and cluster priors imposition.
result Necessary and sufficient conditions for clustering in GANs are identified, and their importance is demonstrated through ablative studies.
PIMA autoencoders discover shared features in multimodal scientific data.
problem Discovering shared information in high-throughput scientific datasets.
method Physics-informed multimodal autoencoders (PIMA) with Gaussian mixture prior and product of experts formulation.
result Accurate cross-modal inference between images and mechanical stress-strain response in lattice metamaterials.
Dictionary learning algorithms have been successfully used for both reconstructive and discriminative tasks, where an input signal is represented with a sparse linear combination of dictionary atoms. While these methods are mostly developed for single-modality scenarios, recent studies have demonstrated the advantages …
Branching Flows generates sequences of varying lengths using binary trees.
problem Generating sequences of unknown lengths or fixed elements.
method A generative modeling framework that evolves states over binary trees, controlling sequence length.
result Branching Flows can generate sequences of varying lengths and mix different types of state spaces.
MMAML adapts quickly to multimodal tasks with task-aware modulation.
problem Limitation of common initialization in MAML for diverse task distributions.
method Augment MAML with task-aware modulation to adapt to multimodal task distributions.
result MMAML improves adaptation efficiency on multimodal few-shot learning tasks.
We introduce a deep multitask architecture to integrate multityped representations of multimodal objects. This multitype exposition is less abstract than the multimodal characterization, but more machine-friendly, and thus is more precise to model. For example, an image can be described by multiple visual views, which …
Graph matching is a challenging problem with very important applications in a wide range of fields, from image and video analysis to biological and biomedical problems. We propose a robust graph matching algorithm inspired in sparsity-related techniques. We cast the problem, resembling group or collaborative sparsity f…
New framework estimates graph from multimodal functional data.
problem Estimating graph from joint multimodal functional data.
method Integrative framework using partial correlation operator.
result Estimator converges to stationary point with quantifiable error.
Paper improves video categorization using temporal coherence.
problem Video categorization in multiple modalities.
method Temporal coherence-based regularization for multimodal models.
result Models with temporal coherence outperform state-of-the-art.
Study quantifies interactions between unlabeled multimodal data.
problem Understanding how modalities combine in semi-supervised settings.
method Information-theoretic definitions and bounds derivation.
result Validated lower and upper bounds accurately track true interactions.