Chain of neural networks learns scale-specific features for fast multimodal image registration.
problem Non-rigid registration of multimodal images, especially in remote sensing.
method Chain of fully-convolutional neural networks designed to learn scale-specific features, predicting deformation directly.
result Global registration in linear time, outperforming current methods in remote sensing tasks.
Improved text-to-image and multimodal understanding through adaptive generation order optimization.
problem Determining optimal generation sequences in text-to-image synthesis and multimodal understanding.
method Introduced a learnable control module trained via Group Relative Policy Optimization (GRPO) to determine the generation order.
result Learning the control block substantially improves text-to-image alignment and multimodal understanding in DLMs.
Paper presents a method to align unpaired samples across different modalities.
problem Challenges in collecting paired samples for multimodal representation learning.
method Uses propensity score alignment based on Rubin's framework to estimate a common space for unpaired samples.
result Optimal transport matching significantly improves alignment in real-world data.
Proposes a copula-driven framework for multimodal learning.
problem Aligning and fusing representations from multiple modalities with complex interactions.
method Copula model for joint distribution of modalities, Gaussian mixture for marginal distributions.
result Superior performance on public MIMIC datasets.
DTA aligns multimodal data with prior correspondence knowledge.
problem Challenges in aligning data from different domains.
method Semi-supervised manifold alignment method exploiting prior correspondence knowledge.
result DTA outperforms other methods in aligning multimodal data.
Graph matching is a challenging problem with very important applications in a wide range of fields, from image and video analysis to biological and biomedical problems. We propose a robust graph matching algorithm inspired in sparsity-related techniques. We cast the problem, resembling group or collaborative sparsity f…
Deep model predicts shapes of curves with multiple covariates.
problem Predicting shapes of planar curves with various covariates.
method Deep learning model using complex-valued functions, conditional covariance smoother with modality-specific encoders.
result Model accurately predicts shapes of curves with multimodal covariates.
A new mathematical framework for multimodal learning.
problem Linking different data modalities for better understanding.
method Interpreting contrastive learning as optimizing encoders for conditional probability distributions.
result Novel probabilistic loss functions and metrics for alignment in latent space.
Hierarchical OT improves alignment of multimodal datasets.
problem Aligning related but noisy datasets.
method Hierarchical optimal transport with clustered structure.
result Hierarchical alignment improves cross-domain data matching.
Multimodal bitransformer boosts image-text classification.
problem Combining text and image modalities for improved classification.
method Supervised multimodal bitransformer model integrating text and image encoders.
result State-of-the-art performance on multimodal classification benchmarks.
DMAN embeds social images with deep multimodal attention networks.
problem Sub-optimal social image representation for social media data.
method Deep Multimodal Attention Networks (DMAN) that jointly embed multimodal contents and link information.
result DMAN achieves significant improvement in multi-label classification and cross-modal search compared to state-of-the-art image embeddings.
Plug-and-play multimodal controller improves class-conditional image generation.
problem Generating class-conditional images from user-specified labels.
method Introduces a `multimodal controller` to generate multimodal data without additional learning parameters.
result Multimodal controlled generative models produce higher quality class-conditional images and novel modalities.
Unified model integrates text and time series for financial forecasting.
problem Challenges in integrating complementary modalities for improved forecasting.
method Modality-specific experts and cross-modal alignment framework.
result State-of-the-art performance on financial forecasting task.
HCL learns shared and modality-specific latent representations for multimodal data.
problem Binary shared-private decomposition inadequately represents shared information across subsets of modalities.
method Hierarchical Contrastive Learning framework combining latent-variable formulation, structural sparsity, and contrastive objective.
result HCL accurately recovers hierarchical structure and improves predictive performance on multimodal data.
Framework improves clinical timeline reconstruction from text and tables.
problem Temporal precision and event timing in clinical narratives and EHRs.
method Retrieval-augmented multimodal alignment framework.
result Consistently improves absolute timestamp accuracy and temporal concordance.
The paper introduces multimodal generative models to improve data marginal likelihood.
problem Improving data marginal likelihood in multimodal settings.
method Derives variational bounds on the evidence for multimodal deep generative models, generalizes objectives for different model types, and benchmarks across various datasets.
result Multimodal VAEs excel in image, label, and text datasets with and without weak supervision.
Model learns multiple tasks using visual and textual representations.
problem Training visual navigation agents for multiple tasks.
method Dual-Attention unit for task-invariant alignment of visual and textual representations.
result Model outperforms baselines on semantic goal navigation and embodied question answering.
Geometric method captures rare topics and temporal alignment in co-author networks.
problem Missing rare topics and smooth temporal alignment in topic modeling.
method Integrates multimodal text and co-author network data using Hellinger distances and Ward's linkage.
result Effective identification of rare topics and visualization of topic drift over time.
A new GAN model LDAGAN uses Latent Dirichlet Allocation to model multimodal images.
problem Ignoring the structure and multimodal characteristics of vision data in GANs.
method Introduced a Dirichlet prior for multimodal image generation leading to LDAGAN. LDAGAN defines generative modes for each sample and uses a VEM algorithm for adversarial training.
result Experimental results show LDAGAN outperforms other GANs on real-world datasets.
Unified framework explains few-shot multimodal medical imaging performance.
problem Limited labeled data in rare diseases and low-resource settings.
method PAC learning, VC theory, PAC Bayesian analysis, information gain, Chain of Thought reasoning.
result Unified theoretical framework for few-shot multimodal medical imaging.
Study analyzes bias interactions in multimodal models using simulation-based methods.
problem Analyzing dynamic bias interactions in multimodal models to ensure fairness and equity.
method Simulation-based heuristic approach to compute bias scores for text-only, image-only, and multimodal embeddings.
result Multimodal bias interactions can be amplification, mitigation, or neutral, with text bias often dominant.
Enhances hand gesture recognition with separate networks and shared features.
problem Improving recognition accuracy of unimodal 3D-CNNs for dynamic hand gestures.
method Separate networks for each modality, collaborative learning, spatiotemporal semantic alignment loss, focal regularization.
result Improves test time recognition accuracy and state-of-the-art performance.
InVA models image outcomes from multiple modalities, outperforming standard VAEs.
problem Understanding relationships across multiple imaging modalities in neuroimaging.
method Integrative Variational Autoencoder (InVA) framework for image-on-image regression.
result InVA accurately predicts PET scans from structural MRI, outperforming conventional models.
Develops a deep multimodal classifier for medical images and reports.
problem Challenges of small datasets in medical deep learning.
method Transfer learning, integrated gradients, unsupervised clustering, tuning parameter.
result Improves classification accuracy by 4% and 7% on average.
VAEs struggle with surjective multimodal data, especially class labels describing images.
problem VAEs struggle to capture variability in surjective multimodal data.
method Theoretical and empirical demonstration of VAEs with a mixture of experts posterior.
result VAEs with a mixture of experts posterior can disregard variation in surjective multimodal data.
Framework translates images between domains without supervision.
problem Challenges in unsupervised image-to-image translation, especially handling multimodality.
method Proposes a Multimodal Unsupervised Image-to-Image Translation (MUNIT) framework, decomposing images into content and style codes.
result Demonstrates improved generation of diverse outputs from a single source image.
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
MKBE embeds multimodal data for knowledge base completion.
problem Missing multimodal data in knowledge bases.
method Multimodal encoders and decoders for text, images, and numerical values.
result State-of-the-art link prediction with 5-7% improvement.
Proposes MR-SNE for multimodal data visualization.
problem Visualizing data from multiple domains with relations across them.
method Extends t-SNE to compute augmented relations and jointly embed them in a low-dimensional space.
result Demonstrates promising performance in visualizing Flickr and Animal with Attributes 2 datasets.
Proposes neural network for causal inference with multimodal data.
problem Estimating causal effects with text and image data as confounders.
method Double machine learning framework adapted to partially linear models, semi-synthetic dataset generation.
result Improved performance in causal effect estimation with multimodal data.
GGMPs improve non-Gaussian conditional density estimation.
problem Multimodality, heteroscedasticity, and strong non-Gaussianity in conditional density estimation.
method GGMP combines local Gaussian mixture fitting, cross-input component alignment, and per-component heteroscedastic GP training.
result GGMPs improve distributional approximation on synthetic and real-world datasets.
Deep neural nets analyze images and text for emotional tagging.
problem Predicting latent emotional states from multimodal data.
method Combining deep neural networks for text and image features.
result Multimodal model outperforms single-modal models.
New method tackles heterogeneous breast cancer imaging data.
problem Challenges in multimodality breast cancer imaging data.
method Developed a multilayer tensor learning method to incorporate heterogeneity.
result Outperforms other existing methods in predicting disease status.
New deep fusion methods improve human action recognition using depth and inertial sensor data.
problem Existing multimodal HAR frameworks lack mid-level feature fusion.
method Proposes three deep multilevel multimodal fusion frameworks, transforming depth and inertial sensor data into images and using convolution with Prewitt filter to create modality within modality.
result Supremacy of proposed fusion frameworks over existing methods on three publicly available datasets.
New method handles missing data in multimodal brain imaging.
problem Missing data in multimodal brain imaging.
method Full Information Linked ICA (FI-LICA) algorithm.
result FI-LICA outperforms current practices in classification and prediction.
Dp-CLIP preserves privacy in multimodal AI training.
problem Privacy concerns in multimodal AI, especially in vision-language tasks.
method Differentially private adaptation of CLIP model.
result Dp-CLIP retains accuracy while ensuring privacy.
OTSeg uses multi-prompt Sinkhorn attention to improve zero-shot semantic segmentation.
problem Leveraging pre-trained CLIP knowledge to align text embeddings with pixel embeddings.
method OTSeg employs Multi-Prompts Sinkhorn (MPS) and Multi-Prompts Sinkhorn Attention (MPSA) to enhance semantic feature matching.
result OTSeg achieves state-of-the-art performance in zero-shot semantic segmentation tasks.
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared …
New results show contrastive learning can recover shared factors in multimodal data.
problem Understanding when contrastive learning can recover shared latent factors in multimodal data.
method New identifiability results for multimodal contrastive learning, distinguishing between multi-view and multimodal settings.
result Contrastive learning can block-identify shared latent factors in multimodal data, even with dependencies.
UniFinEval benchmarks financial models across text, images, and videos.
problem Challenges in evaluating financial multimodal models across text, images, and videos.
method Proposes UniFinEval, a unified multimodal benchmark for financial scenarios.
result Gemini-3-pro-preview achieves best performance but still lags behind experts.
Method retrieves similar fashion items from images and text, enabling style refinement.
problem Lack of intuitive, interactive refinement in search engines for fashion items.
method Joint visual-textual embedding training, Mini-Batch Match Retrieval, attribute extraction.
result Improved performance in multimodal style search, demonstrated through benchmark.
StoryGen uses images to generate coherent text from multiple images.
problem Generating coherent text from multiple images.
method Designing a Relational Text Data Generator algorithm to relate features from plural images.
result The model can generate meaningful paragraphs containing extracted features from input images.
This paper investigates uncertainty calibration in multimodal large language models.
problem Challenges in properly calibrating uncertainty in multimodal large language models.
method Investigation of representative MLLMs across various scenarios, including visual fine-tuning and multimodal training.
result MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments.
Reduces memory costs for developing countries by replacing images with text.
problem High memory costs associated with multimodal webpages in developing countries.
method Canonical Correlation Analysis (CCA) to replace high-cost modality (images) with low-cost modality (text).
result Reduces memory costs by at least 83.35% through eye-tracking experiments.
We introduce a kernel method for manifold alignment (KEMA) and domain adaptation that can match an arbitrary number of data sources without needing corresponding pairs, just few labeled examples in all domains. KEMA has interesting properties: 1) it generalizes other manifold alignment methods, 2) it can align manifold…
Enhances curve alignment for diverse data types.
problem Aligning curve data effectively.
method Developed nonlinear transformations for curve data.
result Successfully aligned synthetic and real curve data.
Improved software flaw detection using NAS on multimodal DL models.
problem Software flaw detection in multimodal deep learning models.
method Adapted NAS framework for multimodal learning, combined with multimodal deep learning models.
result Improved performance on the Juliet Test Suite.
PVAE learns disentangled representations from multimodal data.
problem Learning disentangled representations from multimodal sensory data.
method Partitioned Variational Autoencoder (PVAE) with multimodal generative model and training objectives.
result PVAE achieves over 99% accuracy on both modalities for semantic units.