MHVAE learns cross-modality inference inspired by human cognition.
problem Cross-modality inference in multimodal data.
method Hierarchical multimodal generative model with modality-specific and joint-modality distributions.
result MHVAE performs on par with state-of-the-art models on multimodal datasets.
Cross-modal hashing has been receiving increasing interests for its low storage cost and fast query speed in multi-modal data retrievals. However, most existing hashing methods are based on hand-crafted or raw level features of objects, which may not be optimally compatible with the coding process. Besides, these hashi…
This research shows how to learn shared representations from unpaired data.
problem Learning shared representations from unpaired data.
method Spectral embeddings of random walk matrices from each unimodal representation.
result Shared representations can be learned almost exclusively from unpaired data.
T-EMDE bridges the heterogeneity gap between image and text modalities.
problem Finding similarities between image and text modalities with non-related feature spaces.
method Inspired by EMDE, T-EMDE uses sketches for multimodal operations, avoiding self-attention's quadratic complexity.
result T-EMDE achieves state-of-the-art results and reduces model latency.
New task aligns molecular structure with gene expression changes.
problem Modeling the relationship between chemical structure and gene expression changes.
method Developed a cross-modal small molecule retrieval task and a coordinated deep learning approach to align chemical structure and gene expression profiles.
result Demonstrated the feasibility of the new task and highlighted the limitations of current data and systems.
COBRA reduces modality gap in cross-modal tasks.
problem Joint embedding spaces fail to sufficiently reduce modality gap in multi-modal tasks.
method COBRA trains image and text modalities in a joint fashion using Contrastive Predictive Coding and Noise Contrastive Estimation.
result COBRA significantly reduces the modality gap and generates robust joint-embedding space.
Paper improves video feature learning for better downstream tasks.
problem Improving video feature learning for better performance on downstream tasks.
method Self-supervised learning approach using contrastive bidirectional transformer, extending BERT for real-valued feature vectors.
result Significantly improved performance on video classification, captioning, and segmentation tasks.
The paper analyzes and proposes an algorithm for multi-modal nonlinear embeddings with theoretical performance bounds.
problem Generalizability of multi-modal nonlinear embeddings to unseen data.
method Theoretical analysis and a multi-modal nonlinear representation learning algorithm motivated by performance bounds.
result The proposed algorithm yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
Paper proposes a harmonized approach to multimodal learning using GPLVMs.
problem Modality heterogeneity in multimodal data.
method Develops a novel learning scheme called Harmonization to jointly learn latent model parameters from different modalities.
result Experimental results show superior performance in cross-modal retrieval tasks.
GROOVE learns representations for weakly paired multimodal data.
problem Learning representations for high-content perturbation data with weakly paired samples.
method GroupCLIP contrastive loss integrated with an autoencoder framework.
result GROOVE performs on par with or outperforms existing approaches for cross-modal tasks.
Generative Multisensory Network learns 3D scene representations from multiple modalities.
problem Learning robust 3D scene representations from multiple sensory modalities.
method Amortized Product-of-Experts for efficient inference and cross-modal generation.
result The model can infer modality-invariant 3D scene representations efficiently from various sensory modalities.
Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
Unified model learns joint and individual features from brain imaging data.
problem Integrating structural and functional connectivity data for behavioral phenotypes.
method Cross-Modal Joint-Individual Variational Network (CM-JIVNet) with multi-head attention fusion.
result CM-JIVNet outperforms in cross-modal reconstruction and behavioral trait prediction.
In recent years, there have been numerous developments towards solving multimodal tasks, aiming to learn a stronger representation than through a single modality. Certain aspects of the data can be particularly useful in this case - for example, correlations in the space or time domain across modalities - but should be…
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared …
New test assesses shared brain activity across different cognitive modalities.
problem Determining if different cognitive modalities use overlapping neural representations.
method Formulated a statistical hypothesis testing approach using permutation testing.
result New test (CMPT) has greater statistical power than cross-modal decoding while maintaining low Type I errors.
DNN-based cross-modal retrieval has become a research hotspot, by which users can search results across various modalities like image and text. However, existing methods mainly focus on the pairwise correlation and reconstruction error of labeled data. They ignore the semantically similar and dissimilar constraints bet…
We present a baseline approach for cross-modal knowledge fusion. Different basic fusion methods are evaluated on existing embedding approaches to show the potential of joining knowledge about certain concepts across modalities in a fused concept representation.
AnomalyDAE detects anomalies in networks by learning cross-modality interactions.
problem Detecting anomalies in attributed networks where structure and attributes interact.
method Dual autoencoder framework with attention mechanism for joint learning of structure and attribute embeddings.
result AnomalyDAE effectively detects anomalies by reconstructing node attributes and structures.
A novel cross-modal auto-encoder associates different data types efficiently.
problem Cross-modal data association in heterogeneous datasets.
method Bayesian inference framework with variational auto-encoders and associators.
result Successfully associates visual and auditory data with minimal paired data.
Method learns audio embeddings with contextualized tags.
problem Align audio and tags for cross-modal tasks.
method Audio autoencoder, word embeddings, multi-head self-attention, contrastive loss.
result Multi-head self-attention improves audio representations.
Paper proposes a new unsupervised method for cross-modality data translation without requiring direct mappings.
problem Tackles the challenge of zero-shot cross-modality data translation with fidelity.
method Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion) using score-matching and stochastic diffusion.
result Empirically shows advanced performance compared to other generative models.
CLAREL improves zero-shot learning by using per-image semantic supervision and metric rescaling.
problem Fine-grained cross-modal representation learning for zero-shot classification.
method Instance-based deep metric learning in joint visual and textual space, using per-image semantic supervision and metric rescaling.
result CLAREL consistently outperforms existing approaches on fine-grained zero-shot learning datasets.
Study improves cross-modal bike-share and transit demand prediction.
problem Cross-modal ripple effects in urban transportation demand.
method Transfer learning and stacked LSTM models for cross-modal demand prediction.
result Transfer learning models outperform unimodal models in cross-modal demand prediction.
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
TAP transfers knowledge from unlabeled data to improve cross-modal learning.
problem Improving supervised learning performance using unlabeled data from a different modality.
method Probabilistic approach for missing information estimation, kernel regression, cross-attention module, TAP neural network.
result TAP significantly improves generalization across different domains and neural network architectures.
New method identifies shared components from unpaired multimodal mixtures.
problem Identify shared components from unpaired multimodal mixtures.
method Distribution divergence minimization-based loss with sufficient conditions for identifiability.
result Sufficient conditions for shared component identifiability from unaligned multimodal mixtures.
Efficient video captioning model captures cross-modal interactions.
problem Capturing frame-level cross-modal interactions in video captioning.
method Proposes High-Order Cross-Modal Attention (HOCA) and Low-Rank HOCA.
result Low-Rank HOCA achieves state-of-the-art performance.
Learning social media data embedding by deep models has attracted extensive research interest as well as boomed a lot of applications, such as link prediction, classification, and cross-modal search. However, for social images which contain both link information and multimodal contents (e.g., text description, and visu…
Adaptive anchor methods improve multi-modal learning by balancing intra-modal and inter-modal information.
problem Fixed anchor methods limit multi-modal learning by over-reliance on a single modality and inadequate cross-modal correlation.
method Adaptive anchor methods using centroid-based anchors from all modalities.
result Adaptive anchor methods like CentroBind consistently outperform fixed anchor methods across various datasets.
FlexCMH learns effective hashing codes from weakly-paired data.
problem Cross-modal hashing assumes perfect correspondence between samples, which is unrealistic.
method FlexCMH uses clustering-based matching to find potential correspondence and jointly optimizes it with hashing functions.
result FlexCMH achieves significantly better results than state-of-the-art methods.
Survey on LLMs for time series analytics across various domains.
problem Cross-modality gap between LLMs and time series data.
method Taxonomy of approaches, cross-modality strategies, and experiments on multimodal datasets.
result Effective combinations of textual data and cross-modality strategies enhance time series analytics.
Unified model integrates text and time series for financial forecasting.
problem Challenges in integrating complementary modalities for improved forecasting.
method Modality-specific experts and cross-modal alignment framework.
result State-of-the-art performance on financial forecasting task.
CMTF improves financial market forecasting by fusing multiple data types.
problem Lack of effective integration of diverse financial data sources.
method Transformer-based deep learning framework with tensor interpretation and auto-training.
result CMTF outperforms classical and deep learning models in price direction classification.
TCT learns multimodal sequence representations by translating from related sequences.
problem Challenges in learning semantic representations from multimodalities.
method Transformer based Cross-modal Translator (TCT) combined with Multimodal Transformer Network (MTN).
result Proposed method achieves new state-of-the-art performance on video-grounded dialogue.
SBCA optimizes portfolios by fusing price data and text sentiment.
problem Insufficient integration of multi-modal information in traditional portfolio optimization models.
method Cross-modal BERT-driven Actor-Critic framework with gated fusion and constraint embedding.
result SBCA outperforms benchmarks in portfolio value, return, Sharpe ratio, and maximum drawdown.
Improved sentiment analysis with multimodal data.
problem Cross-modal sentiment analysis in social media, customer service, and video blogs.
method Gated mechanism for attention-based learning of cross-modal interactions, with experiments on CMU-MOSI and CMU-MOSEI datasets.
result 1.6% and 1.34% absolute improvement over state-of-the-art.
Successful multimodal search and retrieval requires the automatic understanding of semantic cross-modal relations, which, however, is still an open research problem. Previous work has suggested the metrics cross-modal mutual information and semantic correlation to model and predict cross-modal semantic relations of ima…
A method uses autoencoders to align multi-modal neuron data.
problem Inconsistent cell type definitions across different data modalities.
method Coupled training of autoencoders for cross-modal alignment.
result Representations learned by coupled autoencoders can identify single-modality sampled cell types.
New methods estimate point-wise dependency from neural MI models.
problem Estimating point-wise dependency between different events.
method Developed two methods: Probabilistic Classifier and Density-Ratio Fitting.
result Demonstrated effectiveness in MI estimation, self-supervised representation learning, and cross-modal retrieval.
Paper tackles cross-modal anomalies in multi-source data.
problem Detect anomalies in multi-modal data where patterns are inconsistent across different sources.
method Proposes a deep structured anomaly detection framework.
result Demonstrates effectiveness on real-world datasets.
We analyse multimodal time-series data corresponding to weight, sleep and steps measurements. We focus on predicting whether a user will successfully achieve his/her weight objective. For this, we design several deep long short-term memory (LSTM) architectures, including a novel cross-modal LSTM (X-LSTM), and demonstra…
Proposes a method to align language and image data.
problem Aligning language and image data for better understanding.
method Uses triplet loss to learn consistent embeddings of language and images.
result Outperforms four baselines across multiple metrics.
Metric-based meta-learning techniques have successfully been applied to few-shot classification problems. In this paper, we propose to leverage cross-modal information to enhance metric-based few-shot learning methods. Visual and semantic feature spaces have different structures by definition. For certain concepts, vis…
Knowledge Transfer (KT) techniques tackle the problem of transferring the knowledge from a large and complex neural network into a smaller and faster one. However, existing KT methods are tailored towards classification tasks and they cannot be used efficiently for other representation learning tasks. In this paper a n…
GEM learns a manifold for cross-modal data, capturing structure without modality dependence.
problem Modality-specific neural models limit flexibility and custom architecture.
method Casts learning as manifold inference, enforcing coverage, linearity, and isometry.
result GEM learns latent structure across image, shape, audio, and cross-modal domains.
Supervised cross-modal hashing has gained increasing research interest on large-scale retrieval task owning to its satisfactory performance and efficiency. However, it still has some challenging issues to be further studied: 1) most of them fail to well preserve the semantic correlations in hash codes because of the la…
Deep learning model improves medical diagnosis and reduces patient time.
problem Ineffective and slow methods for synthesizing multimodal medical data.
method Cross-modal deep learning architecture with co-attention mechanism.
result Model outperforms previous methods by 2.35% and is 53% more efficient.