DMAN embeds social images with deep multimodal attention networks.
problem Sub-optimal social image representation for social media data.
method Deep Multimodal Attention Networks (DMAN) that jointly embed multimodal contents and link information.
result DMAN achieves significant improvement in multi-label classification and cross-modal search compared to state-of-the-art image embeddings.
Paper presents a fast method to generate multimodal embeddings.
problem Integrating visual and linguistic information into a single representation.
method Learning a language-to-vision mapping to build multimodal embeddings.
result Mapped vectors outperform unimodal and multimodal baselines, especially in zero-shot settings.
MKBE embeds multimodal data for knowledge base completion.
problem Missing multimodal data in knowledge bases.
method Multimodal encoders and decoders for text, images, and numerical values.
result State-of-the-art link prediction with 5-7% improvement.
Simple baselines improve multimodal utterance learning.
problem Learning rich multimodal utterance representations.
method Conditional factorization of utterances into unimodal factors; extending to bimodal and trimodal factors.
result Optimal embeddings can be derived in closed form.
Proposes MR-SNE for multimodal data visualization.
problem Visualizing data from multiple domains with relations across them.
method Extends t-SNE to compute augmented relations and jointly embed them in a low-dimensional space.
result Demonstrates promising performance in visualizing Flickr and Animal with Attributes 2 datasets.
Deep learning predicts mental disorders from audio and text samples.
problem Predicting mental disorders from speech samples.
method Multimodal deep learning structure using various pre-trained models for audio and text embeddings, transfer learning, and auxiliary corpora.
result Acceptable accuracy in predicting mental disorders through multimodal analysis.
New embeddings for medical concepts from multimodal data surpass previous methods.
problem Lack of comprehensive embeddings for medical concepts.
method Combining large datasets of medical notes, claims, and journal articles to learn embeddings.
result cui2vec embeddings achieve state-of-the-art performance.
New word distributions capture multiple meanings and outperform existing methods.
problem Capturing semantic information for words with multiple meanings.
method Gaussian mixtures with an energy-based max-margin objective.
result Multimodal word distributions outperform word2vec and Gaussian embeddings.
Paper proposes ICCN to learn correlations between text, audio, and video for multimodal sentiment analysis.
problem Improving multimodal sentiment analysis by learning hidden correlations between text and audio/video features.
method Interaction Canonical Correlation Network (ICCN) using deep canonical correlation analysis (DCCA).
result Empirical results confirm the effectiveness of ICCN in capturing useful information from all three views.
New model improves sentiment analysis by fusing words from audio and video.
problem Improving sentiment analysis in noisy multimodal data.
method Gated Multimodal Embedding LSTM with Temporal Attention.
result State-of-the-art sentiment classification and regression results on CMU-MOSI dataset.
DECAT framework evaluates multimodal models for shared biology, detecting confounders and false positives.
problem Determining if multimodal models learn shared biology or just confounders.
method DECAT framework classifies multimodal representations into four diagnostic scenarios using null-referenced metrics.
result DECAT detects confounders and false positives in multimodal models, improving with larger cohorts and stronger representations.
New method for multi-modal depth prediction challenges.
problem Challenges in applying unimodal coreset selection to multi-modal data.
method Adapted state-of-the-art coreset selection technique for multimodal data.
result Challenges in extending unimodal algorithms to multi-modal scenarios.
Method retrieves similar fashion items from images and text, enabling style refinement.
problem Lack of intuitive, interactive refinement in search engines for fashion items.
method Joint visual-textual embedding training, Mini-Batch Match Retrieval, attribute extraction.
result Improved performance in multimodal style search, demonstrated through benchmark.
MDNE embeds network structures and attributes for better analysis.
problem Preserving both structural and attribute features in network embedding.
method Multimodal Deep Network Embedding (MDNE) using deep model with multiple layers of non-linear functions.
result MDNE outperforms baselines on various tasks with real-world datasets.
New algorithms improve semantic similarity in mobile datasets by incorporating context.
problem Traditional models fail to capture external context in mobile datasets.
method Developed new algorithms to integrate external features into sentence embeddings and semantic similarity scores.
result Applying PCA with eight components and appending multimodal features improves semantic understanding in tweets.
Neural model learns company embeddings from data and news.
problem Subjective industry classification schemes in finance.
method Multimodal neural model training company embeddings.
result Objective company representations capture nuanced relationships.
Paper explores multimodal methods for detecting urban micro-events.
problem Detecting urban micro-events in cities with limited geographical coverage.
method Exploring multimodal fusion methods including early, late, hybrid fusion and representation learning.
result Multimodal approach yields higher performance than unimodal alternatives.
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.
Study analyzes bias interactions in multimodal models using simulation-based methods.
problem Analyzing dynamic bias interactions in multimodal models to ensure fairness and equity.
method Simulation-based heuristic approach to compute bias scores for text-only, image-only, and multimodal embeddings.
result Multimodal bias interactions can be amplification, mitigation, or neutral, with text bias often dominant.
Enhances multimodal generation with Normalizing Flows and correlation analysis.
problem Generating coherent cross-modal data from multiple sources.
method Uses Deep Canonical Correlation Analysis for shared information, Normalizing Flows for diversity, and Product of Experts for scalability.
result Improves likelihood, diversity, and coherence in conditional generation.
Deep RL model uses multimodal data for better stock portfolio optimization.
problem Optimizing trading strategies for SP100 stocks using complex data sources.
method Multimodal deep reinforcement learning with state tensors, CNNs, and RNNs.
result Agent outperforms standard benchmarks in portfolio performance.
PIMA autoencoders discover shared features in multimodal scientific data.
problem Discovering shared information in high-throughput scientific datasets.
method Physics-informed multimodal autoencoders (PIMA) with Gaussian mixture prior and product of experts formulation.
result Accurate cross-modal inference between images and mechanical stress-strain response in lattice metamaterials.
Paper tackles embedding attributed sequences in unsupervised learning.
problem Mining tasks over attributed sequences with dependencies between sequences and attributes.
method Proposes a deep multimodal learning framework, NAS, for unsupervised learning of attributed sequences.
result NAS produces task-independent embeddings for various mining tasks on real-world datasets.
PNNs improve treatment outcomes in TAVR and liver trauma.
problem Improving treatment outcomes in medical procedures.
method Multimodal Prescriptive Neural Networks (PNNs) combining optimization and machine learning.
result PNNs significantly improve estimated outcomes in medical procedures.
STE algorithm predicts human behavior from sensor data.
problem Predicting human behavior from noisy, incomplete, and heterogeneous bio-behavioral data.
method Supervised Tensor Embedding (STE) algorithm for high-dimensional multimodal data.
result Features selection improves prediction performance.
Develops a deep multimodal classifier for medical images and reports.
problem Challenges of small datasets in medical deep learning.
method Transfer learning, integrated gradients, unsupervised clustering, tuning parameter.
result Improves classification accuracy by 4% and 7% on average.
Proposes JK-EGW for multimodal alignment using shared latent space.
problem Aligning data from multiple modalities into a shared representation space.
method Joint kernel entropic Gromov--Wasserstein Optimal Transport (JK-EGW).
result Improved multimodal retrieval performance compared to baselines.
Graph-based multimodal federated learning for HAR improves accuracy and privacy.
problem Challenges in HAR due to noisy data, incomplete measurements, and privacy concerns.
method Proposes GraMFedDHAR, a Graph-based Multimodal Federated Learning framework for HAR tasks, using modality-specific graphs, residual GCNs, and attention-based fusion.
result Experimental results show up to 13 percent performance improvement for MultiModalGCN under differential privacy constraints.
Study proposes a multimodal model for cardiovascular risk prediction using EHRs.
problem Lack of comprehensive risk prediction from EHRs due to unstructured text.
method Proposes a multimodal BiLSTM model integrating structured and unstructured EHR data.
result Proposed BiLSTM model outperforms other DNN architectures in cardiovascular risk prediction.
Low-dimensional vectors improve semantic understanding of music and language.
problem Noise in shared semantics due to individual brain biases.
method Jointly model multiple brains to learn low-dimensional vector embeddings.
result These embeddings outperform high-dimensional fMRI data in music and language classification.
OTSeg uses multi-prompt Sinkhorn attention to improve zero-shot semantic segmentation.
problem Leveraging pre-trained CLIP knowledge to align text embeddings with pixel embeddings.
method OTSeg employs Multi-Prompts Sinkhorn (MPS) and Multi-Prompts Sinkhorn Attention (MPSA) to enhance semantic feature matching.
result OTSeg achieves state-of-the-art performance in zero-shot semantic segmentation tasks.
A new model for estimating multivariate densities efficiently.
problem Estimating complex multivariate densities efficiently.
method CDO model based on kernel mean embeddings and RKHS.
result Competitive performance with neural models and Gaussian processes.
DAIF learns fusion structure from data to improve multimodal supervised learning.
problem Tackles the challenge of determining optimal fusion granularity across heterogeneous data sources.
method Combines random matrix theory and non-parametric dependence measures to learn fusion structure directly from data.
result DAIF outperforms state-of-the-art techniques in predicting T-cell differentiation marker protein expression and patient survival.
PreBit predicts Bitcoin price movements using social media and financial data.
problem Predicting extreme price movements of Bitcoin due to its volatility and speculative trading.
method Hybrid model combining FinBERT embeddings of Twitter content with candlestick data and technical indicators.
result The hybrid model can predict significant market movements with a profitable trading strategy.
Generative models unify heterogeneous data for multimodal fusion.
problem Learning effective representations of mixed numerical and categorical data.
method Bayesian approach with exponential family distributions and Laplace-Bernstein approximation.
result Generative models enable fusion of multimodal data for various machine learning tasks.
Paper tackles zero-shot activity recognition using video features and text embeddings.
problem Zero-shot activity recognition with videos.
method Auto-encoder model for multimodal joint embedding, 3D convolutional action recognition for visual features, GloVe word embeddings for textual features.
result Improved zero-shot recognition results with top-n accuracy and mean Nearest Neighbor Overlap.
Paper proposes Seq2Seq models for multimodal sentiment analysis.
problem Learning representations from multiple modalities in machine learning.
method Two unsupervised Seq2Seq models for multimodal sentiment analysis.
result Seq2Seq models improve F1 Score by twelve points in Bimodal sentiment analysis.
Paper proposes a method to preserve multimodal sentiment analysis fidelity.
problem Lack of fidelity in multimodal fusion for sentiment analysis.
method Variational autoencoder-based approach for modality fusion.
result Empirically shows superior performance over state-of-the-art methods.
AI-enhanced product embeddings boost demand analysis accuracy.
problem Traditional demand analysis struggles with nuanced product attributes.
method Combining text, images, and tabular data with transformer embeddings for causal inference.
result AI-enhanced embeddings improve sales rank and price predictions.
TCT learns multimodal sequence representations by translating from related sequences.
problem Challenges in learning semantic representations from multimodalities.
method Transformer based Cross-modal Translator (TCT) combined with Multimodal Transformer Network (MTN).
result Proposed method achieves new state-of-the-art performance on video-grounded dialogue.
Efficient multimodal fusion reduces complexity and improves performance.
problem Multimodal data fusion with tensor transformations.
method Low-rank Multimodal Fusion using tensors.
result Significant reduction in computational complexity with competitive performance.
Paper proposes RMFN for multimodal language analysis.
problem Modeling interactions between language, visual, and acoustic modalities.
method Recurrent Multistage Fusion Network (RMFN) decomposes fusion into stages focusing on subsets of multimodal signals.
result RMFN achieves state-of-the-art performance across multimodal sentiment analysis, emotion recognition, and speaker traits recognition datasets.
Multimodal bitransformer boosts image-text classification.
problem Combining text and image modalities for improved classification.
method Supervised multimodal bitransformer model integrating text and image encoders.
result State-of-the-art performance on multimodal classification benchmarks.
UR-FUNNY dataset aids in understanding multimodal humor.
problem Understanding humor in a multimodal context is understudied.
method Developed a multimodal dataset (UR-FUNNY) for humor detection.
result UR-FUNNY opens the door to multimodal humor detection research.
Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
This paper proposes a model to learn multimodal representations robust to missing data.
problem Learning multimodal representations from heterogeneous sources of information.
method Optimizes a joint generative-discriminative objective across multimodal data and labels, factorizing representations into multimodal discriminative and modality-specific generative factors.
result The proposed model achieves state-of-the-art performance on six multimodal datasets and can reconstruct missing modalities without significant performance drop.
FMT model improves multimodal sequential learning across language, vision, and acoustic data.
problem Modeling spatio-temporal dynamics across multiple modalities.
method Factorized Multimodal Transformer (FMT) that models intramodal and intermodal dynamics in a factorized manner.
result FMT outperforms existing models on 3 datasets and 21 labels, setting new state of the art.
NewsNet-SDF uses deep learning to integrate financial news with financial data for better asset pricing.
problem Combining unstructured text with structured financial data for accurate asset pricing.
method Adversarial networks and pretrained language model embeddings.
result Substantially outperforms alternatives with a Sharpe ratio of 2.80.