A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New algorithm tackles multimodal bandits with optimal regret bounds.
Improved text-to-image and multimodal understanding through adaptive generation order optimization.
In this paper, we propose a deep multimodal fusion network to fuse multiple modalities (face, iris, and fingerprint) for person identification. The proposed deep multimodal fusion algorithm consists of multiple streams of modality-specific Convolutional Neural Networks (CNNs), which are jointly optimized at multiple fe…
PNNs improve treatment outcomes in TAVR and liver trauma.
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) mode…
NVA combines variational posteriors, annealing, and natural-gradient learning for multimodal optimization.
A framework for uncertainty-aware multimodal learning using conformal Shapley intervals.
Paper presents a method to align unpaired samples across different modalities.
A new objective function using Jensen-Shannon divergence improves generative learning from multiple data types.
Dictionary learning algorithms have been successfully used for both reconstructive and discriminative tasks, where an input signal is represented with a sparse linear combination of dictionary atoms. While these methods are mostly developed for single-modality scenarios, recent studies have demonstrated the advantages …
Systematic review of multimodal data challenges and solutions.
Learning social media data embedding by deep models has attracted extensive research interest as well as boomed a lot of applications, such as link prediction, classification, and cross-modal search. However, for social images which contain both link information and multimodal contents (e.g., text description, and visu…
Deep RL model uses multimodal data for better stock portfolio optimization.
PIMA autoencoders discover shared features in multimodal scientific data.
Paper proposes C-STM for multimodal neuroimaging data classification.
Generates multimodal safety-critical scenarios for robustness evaluation of decision-making algorithms.
Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning…
This paper investigates uncertainty calibration in multimodal large language models.
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of j…
Modified PCA algorithm with continual learning preserves features of previous modes for multimode process monitoring.
New method optimizes portfolios by dynamically integrating ESG constraints.
Survey of multimodal deep generative models for diverse data types.
Multimodal fusion is considered a key step in multimodal tasks such as sentiment analysis, emotion detection, question answering, and others. Most of the recent work on multimodal fusion does not guarantee the fidelity of the multimodal representation with respect to the unimodal representations. In this paper, we prop…
Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exp…
Develops a contrastive framework for data-efficient multimodal learning.
New method clusters multimodal data with consistency.
Multimodalities provide promising performance than unimodality in most tasks. However, learning the semantic of the representations from multimodalities efficiently is extremely challenging. To tackle this, we propose the Transformer based Cross-modal Translator (TCT) to learn unimodal sequence representations by trans…
Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent r…
Paper proposes multimodal contrastive learning for EHR data.
TAROT improves data selection for complex multimodal distributions.
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised mult…
PDNS tackles multimodal sampling challenges using proximal point method.
Study quantifies interactions between unlabeled multimodal data.
This study improves stock price prediction using multimodal data.
Study analyzes bias interactions in multimodal models using simulation-based methods.
The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental rese…
Unified model predicts stock and systemic risks from diverse financial data.
Class-conditional generative models are crucial tools for data generation from user-specified class labels. Existing approaches for class-conditional generative models require nontrivial modifications of backbone generative architectures to model conditional information fed into the model. This paper introduces a plug-…
Integrating visual and linguistic information into a single multimodal representation is an unsolved problem with wide-reaching applications to both natural language processing and computer vision. In this paper, we present a simple method to build multimodal representations by learning a language-to-vision mapping and…
Bayesian VFLMSP improves multimodal survival prediction with privacy.
This paper strengthens the computational separation between multimodal and unimodal learning, showing unimodal learning is hard on typical instances.
Enhances gradient-based discrete samplers with parallel tempering for multimodal distributions.
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
With the increasing popularity of video sharing websites such as YouTube and Facebook, multimodal sentiment analysis has received increasing attention from the scientific community. Contrary to previous works in multimodal sentiment analysis which focus on holistic information in speech segments such as bag of words re…
We tackle here the problem of multimodal image non-rigid registration, which is of prime importance in remote sensing and medical imaging. The difficulties encountered by classical registration approaches include feature design and slow optimization by gradient descent. By analyzing these methods, we note the significa…
Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors. Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the field still lacks a large-scale standard …
New results show contrastive learning can recover shared factors in multimodal data.