Proposes LM3FE for multi-modal feature extraction in image classification.
problem High-dimensional features and multi-modal data challenges.
method Large margin multi-modal multi-task feature extraction (LM3FE) framework.
result LM3FE outperforms single-task feature extraction and multi-modal feature extraction.
Inductive graph-based approach for disease classification with incomplete data.
problem Classifying patients with incomplete multi-modal data.
method Multi-modal graph fusion trained end-to-end for node-level classification.
result Outperforms single static graph approach in multi-modal disease classification.
ESE-FN improves elderly activity recognition accuracy.
problem Recognizing individual actions and human-object interactions in elderly activities.
method Exploits multi-modal features from RGB videos and skeleton sequences using ESE attentions and a new Multi-modal Loss.
result ESE-FN achieves best accuracy on ETRI-Activity3D dataset.
A new memory-based fusion layer improves multi-modal deep learning performance.
problem Improving performance of multi-modal deep learning by addressing long-term dependencies.
method Introducing a Memory based Attentive Fusion (MBAF) layer that incorporates both current and long-term dependencies.
result The MBAF layer enhances fusion and improves performance across different modalities and networks.
Unified Bayesian model for multi-modal, small sample size biomedical data classification.
problem Classifying high-dimensional, multi-modal biomedical data with small sample sizes.
method Combines multi-modal data views into a latent space, prunes irrelevant features, and uses dual kernels for small sample size scenarios.
result Outperforms state-of-the-art models and identifies features aligned with existing markers.
System diagnoses Alzheimer's disease from spoken language using multi-modal features.
problem Early diagnosis of Alzheimer's disease from spoken language.
method Classification system based on spoken language using three approaches (N-gram, i-vector, x-vector).
result Accuracy of 83.6% on the cookie picture description task from Pitt Corpus dementia bank.
New algorithms select and rank features from MTS without feature extraction.
problem Feature extraction step for MTS classification.
method Directly computes similarity between time series and assesses cluster structure matching labels.
result Techniques match labels well without feature extraction.
Proposes a novel method for detecting novelty in multi-modal data.
problem Challenges in detecting novelty in high-dimensional, multi-modal data.
method Orthogonalized latent space for disentangling features and defining novelty score.
result Proposed method outperforms state-of-the-art algorithms in novelty detection.
Proposes a method to improve urban spatiotemporal forecasting using multi-modal graph interaction.
problem Improving spatiotemporal forecasting in urban areas using graph convolution networks.
method Develops modality interaction mechanisms for multi-graph convolution networks to reduce generalization error.
result Proposed techniques improve prediction accuracy and model robustness compared to state-of-the-art baselines.
Proposes a multi-modal attention network for better stock price prediction.
problem Predicting future stock movements using historical records and social media.
method Extracts semantic information from social media, estimates credibility, and integrates with numeric features.
result Significantly improved prediction accuracy and trading profits compared to previous methods.
Paper introduces topological eigenvalue theorems for tensor analysis in multi-modal data.
problem Lack of deep understanding of tensor structures in multi-modal data fusion.
method Introduces topological perspective to tensor eigenvalue analysis, linking eigenvalues to topological features.
result Establishes new theorems that enhance understanding of tensor structures in data fusion.
This paper improves sentiment classification by combining text, audio, and video data using DCCA.
problem Improving sentiment classification accuracy using multi-modal data.
method Deep Canonical Correlation Analysis (DCCA) for combining text, audio, and video embeddings.
result One-Step DCCA outperforms current state-of-the-art in multi-modal embedding learning.
Study predicts stock price direction on earnings announcement days using multi-modal deep learning.
problem Predicting stock price movements during earnings announcements is challenging due to market noise and discontinuities.
method Constructed a multi-modal feature space combining fundamental metrics, technical indicators, and sentiment scores from financial news articles. Evaluated LSTM and Transformer models against a baseline.
result Transformer model outperforms LSTM in identifying volatile movements, achieving higher macro F1-score.
Study reduces redundant information in multi-modal datasets.
problem Redundant information increases data size and costs.
method Multimodal framework using entropy, mutual info, Lasso, SHAP, PCA, topic modeling.
result Significant feature space can be pruned with minimal loss in performance.
IndiSeek learns disentangled representations by balancing independence and completeness.
problem Learning disentangled representations with mutual information in multi-modal data.
method Combines independence-enforcing objective with a reconstruction loss that bounds conditional mutual information.
result Demonstrates effectiveness on synthetic data, CITE-seq, and real-world multi-modal benchmarks.
MCC algorithm predicts with partial modalities, outperforming full modalities.
problem Predicting with inconsistent and diverse multi-modal data.
method Instance-oriented Multi-modal Classifier Chains (MCC) algorithm.
result MCC outperforms full modalities in prediction.
MEx dataset benchmarks HAR and multi-modal fusion for exercise quality.
problem Recognizing and evaluating exercise quality for Musculoskeletal Disorders patients.
method Multi-sensor, multi-modal dataset with four sensors (pressure mat, depth camera, accelerometers) for HAR and exercise quality assessment.
result Reference performance for each sensor identified, exposing their strengths and weaknesses.
Paper proposes multi-task learning for multi-modal video Q&A.
problem Expensive to create large-scale datasets for multi-modal video Q&A.
method Composed of three networks: video Q&A, temporal retrieval, and modality alignment.
result State-of-the-art results on TVQA dataset.
This work improves multi-modal generative models by using permutation-invariant neural networks.
problem Improving multi-modal generative models with tighter variational objectives.
method Developed more flexible aggregation schemes based on permutation-invariant neural networks.
result Our variational objective and flexible aggregation models can better approximate the true joint distribution.
Proposes a new model to predict polymer properties by integrating various data types.
problem Inaccurate polymer property prediction due to separate modeling of different data types.
method Multi-modal cascade feature transfer using GCN for chemical structure and molecular descriptors.
result Empirically evaluated model shows higher predictive performance than single-feature approaches.
Proposes a flexible normalization method to handle multi-modal data.
problem Reduced effectiveness of batch normalization in multi-modal distributions.
method Extends normalization to multiple means and variances, detecting data modes on-the-fly.
result Outperforms batch normalization and other methods in various experiments.
A novel multi-modal active learning approach using RL for engagement estimation.
problem Challenges in labeling multi-modal human data for accurate user state estimation.
method Deep reinforcement learning for optimal data selection and multi-modal data fusion.
result The proposed approach outperforms existing methods in engagement estimation.
Study uses sentiment analysis to predict cryptocurrency token returns in virtual reality.
problem Predicting cryptocurrency token returns in virtual reality economies.
method Used BERT for sentiment analysis and developed LSTM models integrating multi-modal features.
result Multi-modal model significantly outperforms price-only baseline in prediction accuracy.
GPCCA integrates multi-modal data with missing values, improving clustering accuracy.
problem Integrating and analyzing multi-modal data with missing values and partial observations.
method Generalized Probabilistic Canonical Correlation Analysis (GPCCA) for unsupervised multi-modal data integration and dimensionality reduction.
result GPCCA outperforms existing methods in capturing essential patterns across modalities and provides robust low-dimensional embeddings.
AI framework diagnoses Parkinson's disease with 100% accuracy.
problem Expertise-demanding medical imaging procedures for Parkinson's disease diagnosis.
method End-to-end, multi-modality diagnosis framework using T1-MRI and 11C-CFT PET.
result 100% accuracy in PD/NL classification.
Improved multi-modal emotion recognition using deep learning.
problem Combining acoustic and text modalities for emotion recognition.
method Proposes a deep learning-based approach to fuse text and acoustic data using SincNet for acoustic features and parallel DCNN and Bi-RNN branches for text processing with cross attention.
result Achieves 3.5% improvement in weighted accuracy compared to existing methods.
Improved robustness in multi-modal sensor fusion with deep learning.
problem Inconsistency in fusion weights leading to poor performance under sensor failures.
method Proposes deep multi-modal sensor fusion architectures with fusion weight regularization and target learning.
result Proposed architectures outperform existing deep learning methods under sensor failures.
SAFE detects fake news by analyzing text and image similarities.
problem Detecting fake news with less focus on text-image similarity.
method SAFE uses neural networks to extract text and visual features, then learns their relationship to predict fake news.
result SAFE effectively recognizes fake news based on text, images, or mismatches.
Stein-Encoder isolates genetic signals in multi-modal biomedical data.
problem Integration of high-dimensional genomic data with clinical data obscures genetic predictive impact.
method White-box supervised framework using Stein's method and residualization.
result Stein-Encoder improves predictive accuracy and reveals specific biological mechanisms.
MTRGL learns temporal correlations from multi-modal data for improved pair trading.
problem Discerning temporal correlations among financial entities.
method Combines time series data and discrete features into a temporal graph, using a memory-based temporal graph neural network.
result MTRGL outperforms traditional methods in temporal graph link prediction and pair trading.
A new hashing method handles complex multi-level labels.
problem Handling complex multi-level labels in cross-modal data retrieval.
method Derives a semantic ranking list from feature and label information, integrates semantic ranking into deep cross-modal hashing.
result RDCMH outperforms other methods in cross-modal retrieval applications.
Paper tackles cross-modal anomalies in multi-source data.
problem Detect anomalies in multi-modal data where patterns are inconsistent across different sources.
method Proposes a deep structured anomaly detection framework.
result Demonstrates effectiveness on real-world datasets.
RDPD uses KD to improve models trained on poor data using rich data from other environments.
problem Deploying models in environments with varying data quality and quantity.
method Knowledge distillation from high-quality, rich data to improve low-quality, poor data models.
result RDPD models consistently outperform baselines and state-of-the-art methods across multiple datasets.
The paper analyzes and proposes an algorithm for multi-modal nonlinear embeddings with theoretical performance bounds.
problem Generalizability of multi-modal nonlinear embeddings to unseen data.
method Theoretical analysis and a multi-modal nonlinear representation learning algorithm motivated by performance bounds.
result The proposed algorithm yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
ROME improves density estimation for multi-modal, non-normal data.
problem Robust multi-modal density estimation in non-normal, highly correlated distributions.
method ROME uses clustering to segment multi-modal data into uni-modal clusters, then combines KDE estimates for each cluster.
result ROME outperforms state-of-the-art methods and is more robust to various distributions.
New architecture for RGB-D object recognition outperforms existing methods.
problem Improving object recognition in RGB-D cameras.
method RCFusion architecture combining RGB and depth information.
result RCFusion outperforms existing methods on standard datasets.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
The paper evaluates samplers on multi-modal targets, focusing on mode separation and recovery.
problem Handling multi-modality in sampling.
method Synthetic experimental setting focusing on mode relative importance recovery.
result Illustrates the challenges and potential of samplers in multi-modality.
FACTM combines FA with correlated topic modeling for structured data integration.
problem Integrating structured data modalities like text and single cell sequencing.
method Bayesian FACTM model combining FA and correlated topic modeling with variational inference.
result FACTM outperforms other methods in identifying clusters in structured data and integrating them with simple modalities.
Study shows agents benefit from hearing in addition to vision.
problem Limited effectiveness of vision-only reinforcement learning agents.
method Used audio as complementary information to visual cues in state representation.
result Agents perform better when hearing is added to vision.
AF improves sampling from high-dimensional, multi-modal distributions.
problem Sampling from high-dimensional, multi-modal distributions is challenging.
method Annealing Flow (AF) using Continuous Normalizing Flow (CNF) with dynamic Optimal Transport (OT) objective and annealing procedures.
result AF significantly improves training efficiency and stability, outperforming state-of-the-art methods.
Radiomics aims to extract and analyze large numbers of quantitative features from medical images and is highly promising in staging, diagnosing, and predicting outcomes of cancer treatments. Nevertheless, several challenges need to be addressed to construct an optimal radiomics predictive model. First, the predictive p…
The group affect or emotion in an image of people can be inferred by extracting features about both the people in the picture and the overall makeup of the scene. The state-of-the-art on this problem investigates a combination of facial features, scene extraction and even audio tonality. This paper combines three addit…
Obtaining common representations from different modalities is important in that they are interchangeable with each other in a classification problem. For example, we can train a classifier on image features in the common representations and apply it to the testing of the text features in the representations. Existing m…
Develops multi-modal neural network models for improved prediction and uncertainty quantification.
problem Improving prediction accuracy and uncertainty quantification for multi-modal data.
method Multi-modal Bayesian neural network models with conjugate last-layer estimation using SVI.
result Improved prediction accuracy and uncertainty quantification compared to uni-modal models.
METEOR learns efficient representations from multi-modal data streams.
problem Efficiently interpreting multi-modal information in complex environments.
method METEOR learns compact representations by sharing parameters within semantically meaningful groups and preserving domain-agnostic semantics.
result METEOR reduces memory usage by around 80% compared to conventional methods.
Framework for handling long-tailed multi-modal data.
problem Class imbalance and long-tailed distributions in multi-modal data.
method Multi-expert architecture with modality-specific networks and dynamic fusion weights.
result Framework outperforms existing methods in long-tailed, class-imbalanced scenarios.
Urban2Vec combines street view imagery and POIs for better urban neighborhood embeddings.
problem Lack of comprehensive representation of urban neighborhoods using heterogeneous data.
method Unsupervised multi-modal framework using CNN for visual features and bag-of-words for POI data.
result Urban2Vec achieves better performance than baseline models and comparable to fully-supervised methods.