UR-FUNNY dataset aids in understanding multimodal humor.
problem Understanding humor in a multimodal context is understudied.
method Developed a multimodal dataset (UR-FUNNY) for humor detection.
result UR-FUNNY opens the door to multimodal humor detection research.
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
FMT model improves multimodal sequential learning across language, vision, and acoustic data.
problem Modeling spatio-temporal dynamics across multiple modalities.
method Factorized Multimodal Transformer (FMT) that models intramodal and intermodal dynamics in a factorized manner.
result FMT outperforms existing models on 3 datasets and 21 labels, setting new state of the art.
MUTLA dataset analyzes multimodal data for teaching and learning analytics.
problem Lack of a comprehensive multimodal dataset for teaching and learning analytics.
method Presented a large-scale MUTLA dataset with synchronized multimodal data from SAIL.
result Provides insights for predicting student engagement and improving adaptive learning.
Paper proposes a method to preserve multimodal sentiment analysis fidelity.
problem Lack of fidelity in multimodal fusion for sentiment analysis.
method Variational autoencoder-based approach for modality fusion.
result Empirically shows superior performance over state-of-the-art methods.
PNNs improve treatment outcomes in TAVR and liver trauma.
problem Improving treatment outcomes in medical procedures.
method Multimodal Prescriptive Neural Networks (PNNs) combining optimization and machine learning.
result PNNs significantly improve estimated outcomes in medical procedures.
Paper proposes MAG to fine-tune BERT and XLNet for multimodal sentiment analysis.
problem Fine-tuning pre-trained models for multimodal language applications is challenging.
method Integrates vision and acoustic modalities into BERT and XLNet through Multimodal Adaptation Gate (MAG).
result Significant improvement in multimodal sentiment analysis performance over previous methods.
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of j…
PIMA autoencoders discover shared features in multimodal scientific data.
problem Discovering shared information in high-throughput scientific datasets.
method Physics-informed multimodal autoencoders (PIMA) with Gaussian mixture prior and product of experts formulation.
result Accurate cross-modal inference between images and mechanical stress-strain response in lattice metamaterials.
Study analyzes bias interactions in multimodal models using simulation-based methods.
problem Analyzing dynamic bias interactions in multimodal models to ensure fairness and equity.
method Simulation-based heuristic approach to compute bias scores for text-only, image-only, and multimodal embeddings.
result Multimodal bias interactions can be amplification, mitigation, or neutral, with text bias often dominant.
Plug-and-play multimodal controller improves class-conditional image generation.
problem Generating class-conditional images from user-specified labels.
method Introduces a `multimodal controller` to generate multimodal data without additional learning parameters.
result Multimodal controlled generative models produce higher quality class-conditional images and novel modalities.
This paper addresses the problem of learning dictionaries for multimodal datasets, i.e. datasets collected from multiple data sources. We present an algorithm called multimodal sparse Bayesian dictionary learning (MSBDL). MSBDL leverages information from all available data modalities through a joint sparsity constraint…
Open-FinLLMs tackle financial tasks with multimodal capabilities.
problem Financial LLMs lack multimodal capabilities and real-world applicability.
method Developed Open-FinLLMs, an open-source multimodal financial LLM suite.
result Open-FinLLMs outperform advanced financial and general LLMs in diverse tasks.
GWN improves multimodal data fusion accuracy for chronic pain patients.
problem Dynamic and unspecified uncertainties in multimodal data fusion.
method Inspired by Global Workspace Theory, GWN is a neural network architecture that dynamically attends to multiple modalities.
result GWN achieved higher F1 scores (0.92 and 0.75) for multimodal discrimination and classification tasks.
Improved sentiment analysis with multimodal data.
problem Cross-modal sentiment analysis in social media, customer service, and video blogs.
method Gated mechanism for attention-based learning of cross-modal interactions, with experiments on CMU-MOSI and CMU-MOSEI datasets.
result 1.6% and 1.34% absolute improvement over state-of-the-art.
Integrates MRF into multimodal VAE for better complex intermodal interactions.
problem Lack of effective modeling of complex intermodal interactions in multimodal VAEs.
method Incorporates Markov Random Field into prior and posterior distributions of multimodal VAE.
result Demonstrates superior performance in managing complex intermodal dependencies.
Develops a contrastive framework for data-efficient multimodal learning.
problem Expensive training of multimodal generative models requiring related multimodal data.
method Contrastive framework for multimodal learning, distinguishing related from unrelated data.
result Data-efficient multimodal learning on challenging datasets for various VAE models.
EmbraceNet improves robustness in multimodal classification.
problem Robust multimodal classification with partial data loss.
method Deep learning architecture for multimodal fusion.
result EmbraceNet outperforms other models in partial data scenarios.
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. Th…
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) mode…
Characterizing the dynamic interactive patterns of complex systems helps gain in-depth understanding of how components interrelate with each other while performing certain functions as a whole. In this study, we present a novel multimodal data fusion approach to construct a complex network, which models the interaction…
New model learns multimodal data better than DAGs.
problem Complex multimodal data not well captured by DAGs.
method Latent partial causal model with two latent coupled variables.
result Identifiability result shows representations correspond to latent variables.
Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches, however, primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in knowledge bases, such as text, …
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared …
We introduce Disease Knowledge Transfer (DKT), a novel technique for transferring biomarker information between related neurodegenerative diseases. DKT infers robust multimodal biomarker trajectories in rare neurodegenerative diseases even when only limited, unimodal data is available, by transferring information from …
We present convolutional neural network (CNN) based approaches for unsupervised multimodal subspace clustering. The proposed framework consists of three main stages - multimodal encoder, self-expressive layer, and multimodal decoder. The encoder takes multimodal data as input and fuses them to a latent space representa…
Estimates interactions between modalities for multimodal data.
problem Accurately quantifying interactions between different data types.
method Developed Lightweight Sample-wise Multimodal Interaction (LSMI) estimator using pointwise information theory.
result LSMI reveals fine-grained dynamics in multimodal data.
Paper proposes a harmonized approach to multimodal learning using GPLVMs.
problem Modality heterogeneity in multimodal data.
method Develops a novel learning scheme called Harmonization to jointly learn latent model parameters from different modalities.
result Experimental results show superior performance in cross-modal retrieval tasks.
Hierarchical OT improves alignment of multimodal datasets.
problem Aligning related but noisy datasets.
method Hierarchical optimal transport with clustered structure.
result Hierarchical alignment improves cross-domain data matching.
Proposes neural network for causal inference with multimodal data.
problem Estimating causal effects with text and image data as confounders.
method Double machine learning framework adapted to partially linear models, semi-synthetic dataset generation.
result Improved performance in causal effect estimation with multimodal data.
This paper investigates uncertainty calibration in multimodal large language models.
problem Challenges in properly calibrating uncertainty in multimodal large language models.
method Investigation of representative MLLMs across various scenarios, including visual fine-tuning and multimodal training.
result MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments.
MISA combines multiple datasets for better feature extraction.
problem Combining diverse datasets for better feature extraction.
method MISA combines multiple heterogeneous datasets using Kotz distribution and combinatorial optimization.
result MISA produces robust generalization of ICA, IVA, and ISA.
Paper explores multimodal methods for detecting urban micro-events.
problem Detecting urban micro-events in cities with limited geographical coverage.
method Exploring multimodal fusion methods including early, late, hybrid fusion and representation learning.
result Multimodal approach yields higher performance than unimodal alternatives.
New results show contrastive learning can recover shared factors in multimodal data.
problem Understanding when contrastive learning can recover shared latent factors in multimodal data.
method New identifiability results for multimodal contrastive learning, distinguishing between multi-view and multimodal settings.
result Contrastive learning can block-identify shared latent factors in multimodal data, even with dependencies.
New deep fusion methods improve human action recognition using depth and inertial sensor data.
problem Existing multimodal HAR frameworks lack mid-level feature fusion.
method Proposes three deep multilevel multimodal fusion frameworks, transforming depth and inertial sensor data into images and using convolution with Prewitt filter to create modality within modality.
result Supremacy of proposed fusion frameworks over existing methods on three publicly available datasets.
Data is one of the essential ingredients to power deep learning research. Small datasets, especially specific to medical institutes, bring challenges to deep learning training stage. This work aims to develop a practical deep multimodal that can classify patients into abnormal and normal categories accurately as well a…
With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words re…
PLS-SVD struggles with missing data in multimodal datasets, showing a phase transition in performance.
problem Missing data in PLS-SVD for multimodal datasets.
method Replica-symmetric analysis of spiked rectangular random matrices with missing entries.
result PLS-SVD performance transitions from uninformative to informative singular vectors at a critical signal-to-noise threshold.
The majority of biomedical studies use limited datasets that may not generalize over large heterogeneous datasets that have been collected over several decades. The current paper develops and validates several multimodal models that can predict 1-year mortality based on a massive clinical dataset. Our focus on predicti…
FinAgent tackles financial trading with multimodal data and advanced AI.
problem Challenges in handling multimodal financial data and limited generalizability.
method Multimodal foundational agent with tool augmentation, dual-level reflection, and diversified memory retrieval.
result Significantly outperforms state-of-the-art baselines in financial trading tasks.
nuScenes dataset includes multimodal sensor data for autonomous vehicle training.
problem Training robust detection and tracking methods for autonomous vehicles.
method Presented the first multimodal dataset with 6 cameras, 5 radars, and 1 lidar, 360-degree field of view.
result 7x more annotations and 100x more images than KITTI dataset.
GGMPs improve non-Gaussian conditional density estimation.
problem Multimodality, heteroscedasticity, and strong non-Gaussianity in conditional density estimation.
method GGMP combines local Gaussian mixture fitting, cross-input component alignment, and per-component heteroscedastic GP training.
result GGMPs improve distributional approximation on synthetic and real-world datasets.
Study improves stock movement prediction using multimodal data.
problem Inaccurate stock movement prediction due to incomplete multimodal data integration.
method Introduces MSGCA framework for robust multimodal fusion.
result MSGCA framework outperforms existing methods by 21.7% on multimodal datasets.
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
We propose a multimodal approach for detection of students' behavioral engagement states (i.e., On-Task vs. Off-Task), based on three unobtrusive modalities: Appearance, Context-Performance, and Mouse. Final behavioral engagement states are achieved by fusing modality-specific classifiers at the decision level. Various…
Proposes MR-SNE for multimodal data visualization.
problem Visualizing data from multiple domains with relations across them.
method Extends t-SNE to compute augmented relations and jointly embed them in a low-dimensional space.
result Demonstrates promising performance in visualizing Flickr and Animal with Attributes 2 datasets.
Benchmarking AutoML for tables with text fields, achieving top performance.
problem Evaluating automated learning systems for tables with text fields.
method Publicly available benchmark with 18 datasets varying in size, types, and feature composition.
result Stack ensembling a multimodal Transformer with various tree models achieved top performance.