CMTF improves financial market forecasting by fusing multiple data types.
problem Lack of effective integration of diverse financial data sources.
method Transformer-based deep learning framework with tensor interpretation and auto-training.
result CMTF outperforms classical and deep learning models in price direction classification.
TAP transfers knowledge from unlabeled data to improve cross-modal learning.
problem Improving supervised learning performance using unlabeled data from a different modality.
method Probabilistic approach for missing information estimation, kernel regression, cross-attention module, TAP neural network.
result TAP significantly improves generalization across different domains and neural network architectures.
Paper improves video feature learning for better downstream tasks.
problem Improving video feature learning for better performance on downstream tasks.
method Self-supervised learning approach using contrastive bidirectional transformer, extending BERT for real-valued feature vectors.
result Significantly improved performance on video classification, captioning, and segmentation tasks.
DNN-based cross-modal retrieval has become a research hotspot, by which users can search results across various modalities like image and text. However, existing methods mainly focus on the pairwise correlation and reconstruction error of labeled data. They ignore the semantically similar and dissimilar constraints bet…
A new hashing method handles complex multi-level labels.
problem Handling complex multi-level labels in cross-modal data retrieval.
method Derives a semantic ranking list from feature and label information, integrates semantic ranking into deep cross-modal hashing.
result RDCMH outperforms other methods in cross-modal retrieval applications.
A novel cross-modal auto-encoder associates different data types efficiently.
problem Cross-modal data association in heterogeneous datasets.
method Bayesian inference framework with variational auto-encoders and associators.
result Successfully associates visual and auditory data with minimal paired data.
New method identifies shared components from unpaired multimodal mixtures.
problem Identify shared components from unpaired multimodal mixtures.
method Distribution divergence minimization-based loss with sufficient conditions for identifiability.
result Sufficient conditions for shared component identifiability from unaligned multimodal mixtures.
Paper proposes a new unsupervised method for cross-modality data translation without requiring direct mappings.
problem Tackles the challenge of zero-shot cross-modality data translation with fidelity.
method Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion) using score-matching and stochastic diffusion.
result Empirically shows advanced performance compared to other generative models.
Study improves cross-modal bike-share and transit demand prediction.
problem Cross-modal ripple effects in urban transportation demand.
method Transfer learning and stacked LSTM models for cross-modal demand prediction.
result Transfer learning models outperform unimodal models in cross-modal demand prediction.
Predicts whether an image is an abstraction of text or vice versa.
problem Understanding semantic cross-modal relations between images and text.
method Introduces a new metric (ABS) and a deep learning autoencoder approach to predict it.
result Demonstrates feasibility of predicting the relative abstractness level of image-text pairs.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
MHVAE learns cross-modality inference inspired by human cognition.
problem Cross-modality inference in multimodal data.
method Hierarchical multimodal generative model with modality-specific and joint-modality distributions.
result MHVAE performs on par with state-of-the-art models on multimodal datasets.
TCT learns multimodal sequence representations by translating from related sequences.
problem Challenges in learning semantic representations from multimodalities.
method Transformer based Cross-modal Translator (TCT) combined with Multimodal Transformer Network (MTN).
result Proposed method achieves new state-of-the-art performance on video-grounded dialogue.
Efficient video captioning model captures cross-modal interactions.
problem Capturing frame-level cross-modal interactions in video captioning.
method Proposes High-Order Cross-Modal Attention (HOCA) and Low-Rank HOCA.
result Low-Rank HOCA achieves state-of-the-art performance.
FlexCMH learns effective hashing codes from weakly-paired data.
problem Cross-modal hashing assumes perfect correspondence between samples, which is unrealistic.
method FlexCMH uses clustering-based matching to find potential correspondence and jointly optimizes it with hashing functions.
result FlexCMH achieves significantly better results than state-of-the-art methods.
Survey on LLMs for time series analytics across various domains.
problem Cross-modality gap between LLMs and time series data.
method Taxonomy of approaches, cross-modality strategies, and experiments on multimodal datasets.
result Effective combinations of textual data and cross-modality strategies enhance time series analytics.
TerraNova models Earth and societies as a unified system.
problem Modeling the physical Earth and human societies as a coupled system.
method TerraNova integrates physical Earth fields and societal indicators in their native geometries using encoders, cross-modal transformers, and a hypernetwork.
result TerraNova represents the physical Earth and societies without lossy averaging over borders, achieving competitive performance and spanning axes not represented by purpose-built encoders.
Efficient Discrete Supervised Hashing improves cross-modal retrieval by preserving semantic correlations and reducing quantization error.
problem Challenges in preserving semantic correlations and reducing quantization error in cross-modal hashing for large-scale retrieval.
method Collective matrix factorization on heterogenous features and semantic embedding with class labels to learn hash codes efficiently.
result EDSH produces superior performance in both accuracy and scalability over existing methods.
SBCA optimizes portfolios by fusing price data and text sentiment.
problem Insufficient integration of multi-modal information in traditional portfolio optimization models.
method Cross-modal BERT-driven Actor-Critic framework with gated fusion and constraint embedding.
result SBCA outperforms benchmarks in portfolio value, return, Sharpe ratio, and maximum drawdown.
Cross-modal data programming speeds medical machine learning.
problem Labeling medical datasets is time-consuming and requires expert knowledge.
method Generates training labels by writing rules over auxiliary modalities, estimating accuracies and correlations.
result Matches or exceeds hand-labeling with statistical significance, faster and more flexible.
Improved sentiment analysis with multimodal data.
problem Cross-modal sentiment analysis in social media, customer service, and video blogs.
method Gated mechanism for attention-based learning of cross-modal interactions, with experiments on CMU-MOSI and CMU-MOSEI datasets.
result 1.6% and 1.34% absolute improvement over state-of-the-art.
T-EMDE bridges the heterogeneity gap between image and text modalities.
problem Finding similarities between image and text modalities with non-related feature spaces.
method Inspired by EMDE, T-EMDE uses sketches for multimodal operations, avoiding self-attention's quadratic complexity.
result T-EMDE achieves state-of-the-art results and reduces model latency.
Automates X-CNN design with improved accuracy and efficiency.
problem Designing cross-modal convolutional neural networks (X-CNNs) is time-consuming and complex.
method Two approaches: data-driven and iterative, to learn and optimise X-CNN topologies.
result Superior performance and up to 9% accuracy gain compared to hand-designed X-CNNs.
COBRA reduces modality gap in cross-modal tasks.
problem Joint embedding spaces fail to sufficiently reduce modality gap in multi-modal tasks.
method COBRA trains image and text modalities in a joint fashion using Contrastive Predictive Coding and Noise Contrastive Estimation.
result COBRA significantly reduces the modality gap and generates robust joint-embedding space.
New test assesses shared brain activity across different cognitive modalities.
problem Determining if different cognitive modalities use overlapping neural representations.
method Formulated a statistical hypothesis testing approach using permutation testing.
result New test (CMPT) has greater statistical power than cross-modal decoding while maintaining low Type I errors.
Paper tackles cross-modal anomalies in multi-source data.
problem Detect anomalies in multi-modal data where patterns are inconsistent across different sources.
method Proposes a deep structured anomaly detection framework.
result Demonstrates effectiveness on real-world datasets.
Unified model learns joint and individual features from brain imaging data.
problem Integrating structural and functional connectivity data for behavioral phenotypes.
method Cross-Modal Joint-Individual Variational Network (CM-JIVNet) with multi-head attention fusion.
result CM-JIVNet outperforms in cross-modal reconstruction and behavioral trait prediction.
Proposes a method to align language and image data.
problem Aligning language and image data for better understanding.
method Uses triplet loss to learn consistent embeddings of language and images.
result Outperforms four baselines across multiple metrics.
EmbraceNet improves robustness in multimodal classification.
problem Robust multimodal classification with partial data loss.
method Deep learning architecture for multimodal fusion.
result EmbraceNet outperforms other models in partial data scenarios.
Adaptive cross-modal few-shot learning improves performance in image classification.
problem Few-shot classification with limited data.
method Adaptive combination of visual and semantic features.
result Model outperforms uni-modality and modality-alignment methods.
Neural networks fail to bridge modalities as expected.
problem Understanding how well neural networks map between different modalities.
method Proposed a new similarity measure and conducted experiments on cross-modal benchmarks.
result Predicted vectors do not resemble the target vectors' neighborhood structure.
New method predicts antibody paratope with high efficiency and accuracy.
problem Predicting the amino acids in the paratope region of antibodies.
method Leveraging à trous convolutions and self-attention, implementing cross-modal attention.
result New state-of-the-art results in paratope prediction.
GEM learns a manifold for cross-modal data, capturing structure without modality dependence.
problem Modality-specific neural models limit flexibility and custom architecture.
method Casts learning as manifold inference, enforcing coverage, linearity, and isometry.
result GEM learns latent structure across image, shape, audio, and cross-modal domains.
AnomalyDAE detects anomalies in networks by learning cross-modality interactions.
problem Detecting anomalies in attributed networks where structure and attributes interact.
method Dual autoencoder framework with attention mechanism for joint learning of structure and attribute embeddings.
result AnomalyDAE effectively detects anomalies by reconstructing node attributes and structures.
Deep learning model improves medical diagnosis and reduces patient time.
problem Ineffective and slow methods for synthesizing multimodal medical data.
method Cross-modal deep learning architecture with co-attention mechanism.
result Model outperforms previous methods by 2.35% and is 53% more efficient.
The heterogeneity-gap between different modalities brings a significant challenge to multimedia information retrieval. Some studies formalize the cross-modal retrieval tasks as a ranking problem and learn a shared multi-modal embedding space to measure the cross-modality similarity. However, previous methods often esta…
Deep LSTM models predict weight objectives from sleep and step data.
problem Predicting weight objectives from multimodal time-series data.
method Design and optimise cross-modal LSTM architectures, including X-LSTM.
result X-LSTM improves prediction accuracy with fewer parameters.
DMAN embeds social images with deep multimodal attention networks.
problem Sub-optimal social image representation for social media data.
method Deep Multimodal Attention Networks (DMAN) that jointly embed multimodal contents and link information.
result DMAN achieves significant improvement in multi-label classification and cross-modal search compared to state-of-the-art image embeddings.
New task aligns molecular structure with gene expression changes.
problem Modeling the relationship between chemical structure and gene expression changes.
method Developed a cross-modal small molecule retrieval task and a coordinated deep learning approach to align chemical structure and gene expression profiles.
result Demonstrated the feasibility of the new task and highlighted the limitations of current data and systems.
This research shows how to learn shared representations from unpaired data.
problem Learning shared representations from unpaired data.
method Spectral embeddings of random walk matrices from each unimodal representation.
result Shared representations can be learned almost exclusively from unpaired data.
MAESTRO improves multimodal learning for dynamic time series with adaptive attention and robustness.
problem Challenges in multimodal learning, especially in healthcare and daily living.
method Dynamic intra- and cross-modal interactions, symbolic tokenization, adaptive attention budgeting, sparse cross-modal attention, MoE mechanism.
result Average relative improvements of 4% and 8% over existing multimodal and multivariate approaches, respectively, under complete observations.
Improved 3D ECG feature attributions for clinical interpretation.
problem Lack of interpretability in deep learning models for 12-lead ECG analysis.
method Cross-modal mapping of feature attributions from 12-lead ECG models onto CineECG 3D space.
result Mapped feature attributions yield higher Dice scores than standard 12-lead attributions.
The paper analyzes and proposes an algorithm for multi-modal nonlinear embeddings with theoretical performance bounds.
problem Generalizability of multi-modal nonlinear embeddings to unseen data.
method Theoretical analysis and a multi-modal nonlinear representation learning algorithm motivated by performance bounds.
result The proposed algorithm yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
Enhances hashing for cross-modal retrieval using multi-view features.
problem Limited improvement in single-view hashing for cross-modal retrieval.
method Exploits multiple views to enrich feature information, learning discriminative hash codes.
result Superior performance compared to state-of-the-art methods on various datasets.
Paper proposes a harmonized approach to multimodal learning using GPLVMs.
problem Modality heterogeneity in multimodal data.
method Develops a novel learning scheme called Harmonization to jointly learn latent model parameters from different modalities.
result Experimental results show superior performance in cross-modal retrieval tasks.
Unified model integrates text and time series for financial forecasting.
problem Challenges in integrating complementary modalities for improved forecasting.
method Modality-specific experts and cross-modal alignment framework.
result State-of-the-art performance on financial forecasting task.
Unified model predicts stock and systemic risks from diverse financial data.
problem Isolating financial tasks leads to missed cross-scale dependencies.
method Shared Transformer backbone with modular task heads for cross-modal attention and multi-task optimization.
result Uni-FinLLM significantly outperforms baselines in stock forecasting, credit-risk assessment, and systemic-risk detection.
A new framework quantifies how model explanations influence each other.
problem Understanding how different model explanations interact and influence each other.
method Introducing the metagame, a conceptual framework for measuring second-order interaction effects of model explanations using Shapley values.
result Meta-attributions provide directional insights into how feature interactions influence model explanations.