Paper proposes RMFN for multimodal language analysis.
problem Modeling interactions between language, visual, and acoustic modalities.
method Recurrent Multistage Fusion Network (RMFN) decomposes fusion into stages focusing on subsets of multimodal signals.
result RMFN achieves state-of-the-art performance across multimodal sentiment analysis, emotion recognition, and speaker traits recognition datasets.
UR-FUNNY dataset aids in understanding multimodal humor.
problem Understanding humor in a multimodal context is understudied.
method Developed a multimodal dataset (UR-FUNNY) for humor detection.
result UR-FUNNY opens the door to multimodal humor detection research.
lamBERT learns language and actions using multimodal BERT.
problem Learning language and actions in complex environments.
method Extending BERT to multimodal representation and integrating with reinforcement learning.
result lamBERT model achieved higher rewards in multitask and transfer settings.
Model learns tensor representations from imperfect multimodal data.
problem Learning from imperfect multimodal data with noise or missing entries.
method Tensor rank minimization to regularize rank of tensor representations.
result Model effectively learns tensor representations from imperfect data.
FMT model improves multimodal sequential learning across language, vision, and acoustic data.
problem Modeling spatio-temporal dynamics across multiple modalities.
method Factorized Multimodal Transformer (FMT) that models intramodal and intermodal dynamics in a factorized manner.
result FMT outperforms existing models on 3 datasets and 21 labels, setting new state of the art.
Paper proposes ICCN to learn correlations between text, audio, and video for multimodal sentiment analysis.
problem Improving multimodal sentiment analysis by learning hidden correlations between text and audio/video features.
method Interaction Canonical Correlation Network (ICCN) using deep canonical correlation analysis (DCCA).
result Empirical results confirm the effectiveness of ICCN in capturing useful information from all three views.
Integrating visual and linguistic information into a single multimodal representation is an unsolved problem with wide-reaching applications to both natural language processing and computer vision. In this paper, we present a simple method to build multimodal representations by learning a language-to-vision mapping and…
VERA-V uses variational inference to discover vulnerabilities in multimodal vision-language models.
problem Existing methods for jailbreaking vision-language models are brittle, limited, and focus on single attacks.
method VERA-V recasts jailbreak discovery as learning a joint posterior distribution over text-image prompts, using variational inference and three complementary strategies.
result VERA-V consistently outperforms state-of-the-art baselines, achieving up to 53.75% higher attack success rate.
Dp-CLIP preserves privacy in multimodal AI training.
problem Privacy concerns in multimodal AI, especially in vision-language tasks.
method Differentially private adaptation of CLIP model.
result Dp-CLIP retains accuracy while ensuring privacy.
Improved text-to-image and multimodal understanding through adaptive generation order optimization.
problem Determining optimal generation sequences in text-to-image synthesis and multimodal understanding.
method Introduced a learnable control module trained via Group Relative Policy Optimization (GRPO) to determine the generation order.
result Learning the control block substantially improves text-to-image alignment and multimodal understanding in DLMs.
This paper investigates uncertainty calibration in multimodal large language models.
problem Challenges in properly calibrating uncertainty in multimodal large language models.
method Investigation of representative MLLMs across various scenarios, including visual fine-tuning and multimodal training.
result MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments.
Paper proposes MAG to fine-tune BERT and XLNet for multimodal sentiment analysis.
problem Fine-tuning pre-trained models for multimodal language applications is challenging.
method Integrates vision and acoustic modalities into BERT and XLNet through Multimodal Adaptation Gate (MAG).
result Significant improvement in multimodal sentiment analysis performance over previous methods.
RiskLabs uses LLMs to predict financial risks from multimodal data.
problem Financial risk prediction using AI techniques.
method Integrates multimodal financial data (textual, vocal, time series, news) into LLMs for prediction.
result Empirical results show effectiveness in forecasting market volatility and variance.
The paper introduces multimodal generative models to improve data marginal likelihood.
problem Improving data marginal likelihood in multimodal settings.
method Derives variational bounds on the evidence for multimodal deep generative models, generalizes objectives for different model types, and benchmarks across various datasets.
result Multimodal VAEs excel in image, label, and text datasets with and without weak supervision.
Paper proposes Seq2Seq models for multimodal sentiment analysis.
problem Learning representations from multiple modalities in machine learning.
method Two unsupervised Seq2Seq models for multimodal sentiment analysis.
result Seq2Seq models improve F1 Score by twelve points in Bimodal sentiment analysis.
Unified model predicts stock and systemic risks from diverse financial data.
problem Isolating financial tasks leads to missed cross-scale dependencies.
method Shared Transformer backbone with modular task heads for cross-modal attention and multi-task optimization.
result Uni-FinLLM significantly outperforms baselines in stock forecasting, credit-risk assessment, and systemic-risk detection.
Low-dimensional vectors improve semantic understanding of music and language.
problem Noise in shared semantics due to individual brain biases.
method Jointly model multiple brains to learn low-dimensional vector embeddings.
result These embeddings outperform high-dimensional fMRI data in music and language classification.
WebGUM learns web navigation from multimodal data, outperforming previous methods.
problem Limited generalization from domain-specific models in web navigation.
method Instruction-following multimodal agent trained on vision-language foundation models.
result Significant improvement in web navigation performance on benchmarks.
Develops a contrastive framework for data-efficient multimodal learning.
problem Expensive training of multimodal generative models requiring related multimodal data.
method Contrastive framework for multimodal learning, distinguishing related from unrelated data.
result Data-efficient multimodal learning on challenging datasets for various VAE models.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
AudioPaLM combines text and speech models to improve speech processing and translation.
problem Improving speech understanding and generation models.
method Unified multimodal architecture combining text and speech models.
result AudioPaLM significantly outperforms existing speech translation systems.
ECC Analyzer uses LLMs to predict stock volatility from ECCs.
problem Leveraging unstructured ECC data for stock volatility prediction.
method Uses large language models to extract and fuse textual and audio features from ECCs.
result ECC Analyzer outperforms traditional benchmarks in volatility prediction.
This study improves stock price prediction using multimodal data.
problem Improving financial asset price forecasting accuracy.
method Combining candlestick time series and textual news flow data using LSTM and pre-trained models.
result Textual modality reduces MAPE by 55%.
DAM improves cryptocurrency trend forecasting using multimodal data.
problem Simplistic merging of sentiment data in cryptocurrency trend forecasting.
method Dual Attention Mechanism (DAM) integrating financial metrics and sentiment analysis.
result DAM outperforms conventional models by up to 20% in prediction accuracy.
Study improves document processing in banking with multimodal analytics.
problem Raising operational efficiency in banking through document-intensive processes.
method Comparative analysis of text classifiers and multimodal model (LayoutXLM) on company register extracts.
result Incorporating layout information in a model substantially increases performance.
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.
Open-FinLLMs tackle financial tasks with multimodal capabilities.
problem Financial LLMs lack multimodal capabilities and real-world applicability.
method Developed Open-FinLLMs, an open-source multimodal financial LLM suite.
result Open-FinLLMs outperform advanced financial and general LLMs in diverse tasks.
Unified model integrates text and time series for financial forecasting.
problem Challenges in integrating complementary modalities for improved forecasting.
method Modality-specific experts and cross-modal alignment framework.
result State-of-the-art performance on financial forecasting task.
Simple baselines improve multimodal utterance learning.
problem Learning rich multimodal utterance representations.
method Conditional factorization of utterances into unimodal factors; extending to bimodal and trimodal factors.
result Optimal embeddings can be derived in closed form.
Study develops sign recognition system for DHH users.
problem Accessibility of voice-controlled devices for Deaf and Hard-of-Hearing users.
method Multimodal data (RGB video and skeletal data) for sign language recognition using deep learning.
result Validation on GMUASL51 dataset of 12 users and 13107 samples across 51 signs.
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
Branching Flows generates sequences of varying lengths using binary trees.
problem Generating sequences of unknown lengths or fixed elements.
method A generative modeling framework that evolves states over binary trees, controlling sequence length.
result Branching Flows can generate sequences of varying lengths and mix different types of state spaces.
UniFinEval benchmarks financial models across text, images, and videos.
problem Challenges in evaluating financial multimodal models across text, images, and videos.
method Proposes UniFinEval, a unified multimodal benchmark for financial scenarios.
result Gemini-3-pro-preview achieves best performance but still lags behind experts.
Model learns multiple tasks using visual and textual representations.
problem Training visual navigation agents for multiple tasks.
method Dual-Attention unit for task-invariant alignment of visual and textual representations.
result Model outperforms baselines on semantic goal navigation and embodied question answering.
ChatGPT struggles in predicting stock movements, underperforming traditional methods.
problem Predicting stock market movements using ChatGPT.
method Zero-shot analysis of ChatGPT's multimodal stock prediction capabilities.
result ChatGPT underperforms traditional methods and state-of-the-art models in predicting stock movements.
This paper explores how combining quantitative factors and news from LLMs improves stock return prediction.
problem Improving stock return prediction using quantitative factors and news.
method Introduces a fusion learning framework to learn unified representations from factors and LLM-generated newsflow, comparing combination, summation, and attentive methods. Explores mixture models and decoupled training approaches.
result Effective multimodal modeling of factors and news improves stock return prediction and selection.
Multimodal analysis assesses job interview performance and provides feedback.
problem Assessing candidate performance in interviews for professional roles.
method Multimodal analytical framework using video, audio, and text data.
result The proposed methodology achieved promising results in predicting behavioral cues.
MM-DREX adapts LLM experts for financial trading via dynamic routing.
problem Challenges of non-stationary financial markets and static expert designs.
method MM-DREX uses a VLM-powered dynamic router to allocate expert weights and designs heterogeneous trading experts.
result Significantly outperforms 15 baselines across key metrics.
Study enhances cryptocurrency sentiment analysis using TikTok and Twitter data.
problem Lack of comprehensive sentiment analysis in cryptocurrency markets.
method Multimodal analysis of TikTok and Twitter data using large language models.
result TikTok's video sentiment influences speculative assets and short-term trends.
DeepBioD combines feature engineering and learned representations for biodegradability prediction.
problem Limited labeled data for biodegradability prediction.
method Developed a multimodal CNN-MLP neural network that integrates engineered and learned representations.
result DeepBioD achieves an error rate of 0.125, 27% lower than state-of-the-art methods.
Extends V-IP framework to use LLMs for generating task-relevant concepts, improving interpretability and performance.
problem Limited applicability of V-IP to small-scale tasks due to manual data annotation.
method Integrates Foundational Models with Large Language and Multimodal Models to generate and annotate concepts.
result FM+V-IP achieves better test performance with fewer concepts/queries compared to other frameworks.
We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual analysis and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the laten…
Survey of financial foundation models for diverse applications.
problem Challenges in applying general-purpose FMs to financial tasks.
method Review of financial foundation models (FFMs) in three modalities.
result Emergence of FFMs designed specifically for finance.
VLM judges rank well but score poorly; task difficulty and annotation quality affect interval width.
problem VLMs as judges lack reliability indicators in multimodal evaluations.
method Conformal prediction using score-token log-probabilities.
result Evaluation uncertainty is task-dependent, affecting interval width and reliability.
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
ViLT is a faster vision-and-language model without convolution or region supervision.
problem Efficiency and expressive power limitations in current VLP models.
method A convolution-free Vision-and-Language Transformer (ViLT) that processes visual inputs similarly to textual inputs.
result ViLT is up to tens of times faster with competitive or better performance.
Framework improves clinical timeline reconstruction from text and tables.
problem Temporal precision and event timing in clinical narratives and EHRs.
method Retrieval-augmented multimodal alignment framework.
result Consistently improves absolute timestamp accuracy and temporal concordance.