New model recognizes emotions with missing modalities, improving accuracy.
problem Handling missing modalities in emotion recognition.
method Transformer-based architecture with cross-attention and self-attention mechanisms.
result Improvement of 37% in predicting arousal values and 30% in valence values compared to baseline.
A framework for uncertainty-aware multimodal learning using conformal Shapley intervals.
problem Uncertainty and modality level importance in multimodal learning.
method Introduces conformal Shapley intervals to quantify modality level importance and uncertainty.
result Demonstrates meaningful uncertainty quantification and strong predictive performance.
Paper proposes a new framework for robust multi-modal data fusion under uncertainty.
problem Unexpected modality failures in nonlinear non-Gaussian dynamic processes.
method Dynamic model averaging (DMA) based particle filter (PF) algorithm.
result The proposed solution outperforms state-of-the-art methods in experiments.
Proposes a parametric modal regression method using the implicit function theorem.
problem Finding conditional modes for multi-modal conditional distributions.
method Uses the implicit function theorem to develop an objective function for learning a joint function over inputs and targets.
result Empirically demonstrates scalability and effectiveness in learning multi-valued functions and high-dimensional inputs.
GPCCA integrates multi-modal data with missing values, improving clustering accuracy.
problem Integrating and analyzing multi-modal data with missing values and partial observations.
method Generalized Probabilistic Canonical Correlation Analysis (GPCCA) for unsupervised multi-modal data integration and dimensionality reduction.
result GPCCA outperforms existing methods in capturing essential patterns across modalities and provides robust low-dimensional embeddings.
Bayesian SSI improves modal parameter uncertainty in operational systems.
problem Uncertainty in modal parameters due to stochastic operational systems and lack of forcing information.
method Proposes a Bayesian stochastic subspace identification (SSI) algorithm with a hierarchical probabilistic model and two inference schemes (Markov Chain Monte Carlo and variational Bayes).
result Posterior distributions over modal properties are obtained, showing lower variance for mean values coinciding with natural frequencies.
JMVAE fails to generate large modalities, JMVAE-kl and JMVAE-h solve this issue.
problem Bi-directional generation between different modalities using VAEs fails for large modalities.
method Proposed JMVAE-kl and JMVAE-h to prevent collapse of joint representation and bi-directional generation.
result JMVAE-kl and JMVAE-h prevent collapse and generate modalities bi-directionally.
Modern statistical methods find the most common value in data.
problem Finding the most common value in data.
method Modern statistical methods applied to mode estimation.
result Traditional approaches to mode estimation are explored and modern methods are applied to new fields.
Unified model integrates text and time series for financial forecasting.
problem Challenges in integrating complementary modalities for improved forecasting.
method Modality-specific experts and cross-modal alignment framework.
result State-of-the-art performance on financial forecasting task.
COIN++ compresses multiple data types efficiently.
problem Handling diverse data modalities in neural compression.
method Implicit neural representations and modulations quantization.
result Significant compression gains with reduced encoding time.
SBCA optimizes portfolios by fusing price data and text sentiment.
problem Insufficient integration of multi-modal information in traditional portfolio optimization models.
method Cross-modal BERT-driven Actor-Critic framework with gated fusion and constraint embedding.
result SBCA outperforms benchmarks in portfolio value, return, Sharpe ratio, and maximum drawdown.
ICYM2I corrects missingness bias in multimodal learning.
problem Missingness patterns between source and target environments affect multimodal learning performance.
method ICYM2I uses inverse probability weighting to correct missingness bias in predictive performance and information gain.
result ICYM2I improves multimodal learning performance by accounting for missingness.
New clustering method for symbolic data combines leaders and agglomerative approaches.
problem Clustering of symbolic data with modal values.
method Proposed a clustering criterion function and leaders clustering method.
result Efficiently solves clustering problems with large data sets.
Introduces MLM dataset for multitask learning across multiple languages and modalities.
problem Training and evaluating multitask systems on diverse data.
method Develops a new benchmark dataset with semantic data and modalities.
result Demonstrates challenges in generalizing on diverse data.
Novel method for multi-view metric learning in vector-valued kernel spaces.
problem Metric learning for multi-view data with multi-modal structure.
method Convex optimization problems and iterative multi-view metric learning algorithm with Nyström approximation.
result Improved performance on real-world datasets compared to state-of-the-art methods.
Study uses sentiment analysis to predict cryptocurrency token returns in virtual reality.
problem Predicting cryptocurrency token returns in virtual reality economies.
method Used BERT for sentiment analysis and developed LSTM models integrating multi-modal features.
result Multi-modal model significantly outperforms price-only baseline in prediction accuracy.
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.
Study reduces redundant information in multi-modal datasets.
problem Redundant information increases data size and costs.
method Multimodal framework using entropy, mutual info, Lasso, SHAP, PCA, topic modeling.
result Significant feature space can be pruned with minimal loss in performance.
Model clarifies network effects on CVA, revealing significant differences in derivative contract values.
problem Network effects on CVA in financial contracts.
method Developed a model to analyze default probabilities in a network of contracts.
result Network effects can significantly alter CVA values, leading to multi-modal distributions.
Taxonomy of knowledge modalities in RL for better transfer.
problem Tackling the transfer of knowledge in reinforcement learning.
method Systematic analysis of knowledge modalities in RL.
result A taxonomy of knowledge modalities in reinforcement learning.
Study predicts stock price direction on earnings announcement days using multi-modal deep learning.
problem Predicting stock price movements during earnings announcements is challenging due to market noise and discontinuities.
method Constructed a multi-modal feature space combining fundamental metrics, technical indicators, and sentiment scores from financial news articles. Evaluated LSTM and Transformer models against a baseline.
result Transformer model outperforms LSTM in identifying volatile movements, achieving higher macro F1-score.
Enhances mobile context prediction using weakly supervised learning.
problem Power consumption and limited training data for always-on context prediction.
method Weakly supervised learning framework for multi-modal sensing.
result Personalized context prediction model with improved accuracy.
This work tackles uncertainty in multi-agent multi-modal trajectory forecasting.
problem Measuring and ranking uncertainty in multi-agent multi-modal trajectory forecasting.
method Proposes collaborative uncertainty (CU) and a CU-aware regression framework.
result The CU-aware regression framework improves SOTA systems' performances.
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
Study reveals limits of PLS in multi-modal learning with correlated signals.
problem Understanding PLS performance in multi-modal learning with correlated signals.
method Random matrix theory analysis of spiked cross-covariance models.
result Identifies SNR and correlation regimes where PLS fails to recover any signal.
Adaptive anchor methods improve multi-modal learning by balancing intra-modal and inter-modal information.
problem Fixed anchor methods limit multi-modal learning by over-reliance on a single modality and inadequate cross-modal correlation.
method Adaptive anchor methods using centroid-based anchors from all modalities.
result Adaptive anchor methods like CentroBind consistently outperform fixed anchor methods across various datasets.
This study improves stock price prediction using multimodal data.
problem Improving financial asset price forecasting accuracy.
method Combining candlestick time series and textual news flow data using LSTM and pre-trained models.
result Textual modality reduces MAPE by 55%.
FlowVAT improves variational inference for multi-modal distributions.
problem Mode-seeking behavior and collapse in variational inference for complex posteriors.
method Conditional tempering approach for normalizing flow variational inference.
result FlowVAT outperforms traditional and adaptive annealing methods in multi-modal distributions, finding more modes and achieving better ELBO values.
KD-Net transfers knowledge from multi-modal to mono-modal segmentation networks.
problem Limited acquisition of multiple imaging modalities in clinical settings.
method Generalized distillation framework adapted for mono-modal networks.
result The student network outperforms baseline mono-modal networks in brain tumor segmentation.
MHVAE learns cross-modality inference inspired by human cognition.
problem Cross-modality inference in multimodal data.
method Hierarchical multimodal generative model with modality-specific and joint-modality distributions.
result MHVAE performs on par with state-of-the-art models on multimodal datasets.
The paper analyzes and proposes an algorithm for multi-modal nonlinear embeddings with theoretical performance bounds.
problem Generalizability of multi-modal nonlinear embeddings to unseen data.
method Theoretical analysis and a multi-modal nonlinear representation learning algorithm motivated by performance bounds.
result The proposed algorithm yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
A novel deep learning technique combines multiple modalities, improving performance.
problem Challenges in leveraging different modalities due to noise and conflicts.
method Proposes a deep neural network that multiplicatively combines information from different modalities.
result Consistent accuracy improvements on three multimodal classification tasks.
MMVAE learns multi-modal data with shared and private latent spaces.
problem Learning useful representations across multiple data modalities.
method Mixture-of-experts variational autoencoder (MMVAE).
result MMVAE satisfies four criteria for multi-modal learning.
Paper improves video feature learning for better downstream tasks.
problem Improving video feature learning for better performance on downstream tasks.
method Self-supervised learning approach using contrastive bidirectional transformer, extending BERT for real-valued feature vectors.
result Significantly improved performance on video classification, captioning, and segmentation tasks.
COBRA reduces modality gap in cross-modal tasks.
problem Joint embedding spaces fail to sufficiently reduce modality gap in multi-modal tasks.
method COBRA trains image and text modalities in a joint fashion using Contrastive Predictive Coding and Noise Contrastive Estimation.
result COBRA significantly reduces the modality gap and generates robust joint-embedding space.
MCC algorithm predicts with partial modalities, outperforming full modalities.
problem Predicting with inconsistent and diverse multi-modal data.
method Instance-oriented Multi-modal Classifier Chains (MCC) algorithm.
result MCC outperforms full modalities in prediction.
CQL learns conservative Q-functions to improve offline RL performance.
problem Leveraging large, static datasets in reinforcement learning without further interaction.
method Conservative Q-learning (CQL) which learns a conservative Q-function to lower-bound policy values.
result CQL substantially outperforms existing offline RL methods, often achieving 2-5 times higher final returns.
Proposes a novel network for CTR prediction by learning modality-specific and modality-invariant representations.
problem Learning good representation of items from multimodal features in E-commerce is challenging due to redundant information across modalities.
method Introduces a Multimodal Adversarial Representation Network (MARN) that calculates modality-specific weights and learns modality-invariant representations.
result Consistently achieves remarkable improvements over state-of-the-art methods in CTR prediction.
DeMIAN learns common representations from different modalities.
problem Learning common representations from unpaired modalities.
method DeMIAN uses adversarial learning to make modalities invariant.
result DeMIAN achieves better classification accuracy.
JMVAE generates and reconstructs multiple modalities from a joint representation.
problem Generating and reconstructing multiple modalities in a bi-directional manner.
method Joint Multimodal Variational Autoencoder (JMVAE) with additional KL divergence reduction method (JMVAE-kl).
result JMVAE can generate and reconstruct multiple modalities more properly than conventional VAEs.
Improves video search by balancing text and visual modalities.
problem Modality imbalance in video search models, focusing mainly on text matching.
method Proposes MBVR with MS samples and DM to balance modalities.
result Empirically shows significant improvement in modality balance and search effectiveness.
Improves cross-modal retrieval by integrating unlabeled data.
problem Lack of semantic similarity constraints and unlabeled data in cross-modal retrieval.
method Integrates quadruplet ranking loss and semi-supervised contrastive loss in a multi-task learning architecture.
result Boosts cross-modal retrieval accuracy by exploiting unlabeled data.
New method learns robust joint representations by translating between modalities.
problem Learning robust joint representations from noisy or missing modalities.
method Cyclic translations between modalities with cycle consistency loss.
result Achieves state-of-the-art results on multimodal sentiment analysis datasets.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
We explore training an automatic modality tagger. Modality is the attitude that a speaker might have toward an event or state. One of the main hurdles for training a linguistic tagger is gathering training data. This is particularly problematic for training a tagger for modality because modality triggers are sparse for…
Framework for handling long-tailed multi-modal data.
problem Class imbalance and long-tailed distributions in multi-modal data.
method Multi-expert architecture with modality-specific networks and dynamic fusion weights.
result Framework outperforms existing methods in long-tailed, class-imbalanced scenarios.
Paper tackles cross-modal anomalies in multi-source data.
problem Detect anomalies in multi-modal data where patterns are inconsistent across different sources.
method Proposes a deep structured anomaly detection framework.
result Demonstrates effectiveness on real-world datasets.
TAP transfers knowledge from unlabeled data to improve cross-modal learning.
problem Improving supervised learning performance using unlabeled data from a different modality.
method Probabilistic approach for missing information estimation, kernel regression, cross-attention module, TAP neural network.
result TAP significantly improves generalization across different domains and neural network architectures.