In this paper, we propose a deep multimodal fusion network to fuse multiple modalities (face, iris, and fingerprint) for person identification. The proposed deep multimodal fusion algorithm consists of multiple streams of modality-specific Convolutional Neural Networks (CNNs), which are jointly optimized at multiple fe…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “multimodal features”
Paper proposes ICCN to learn correlations between text, audio, and video for multimodal sentiment analysis.
problem Improving multimodal sentiment analysis by learning hidden correlations between text and audio/video features.
method Interaction Canonical Correlation Network (ICCN) using deep canonical correlation analysis (DCCA).
result Empirical results confirm the effectiveness of ICCN in capturing useful information from all three views.
Study improves stock movement prediction using multimodal data.
problem Inaccurate stock movement prediction due to incomplete multimodal data integration.
method Introduces MSGCA framework for robust multimodal fusion.
result MSGCA framework outperforms existing methods by 21.7% on multimodal datasets.
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.
Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensorscs.LG
New deep fusion methods improve human action recognition using depth and inertial sensor data.
problem Existing multimodal HAR frameworks lack mid-level feature fusion.
method Proposes three deep multilevel multimodal fusion frameworks, transforming depth and inertial sensor data into images and using convolution with Prewitt filter to create modality within modality.
result Supremacy of proposed fusion frameworks over existing methods on three publicly available datasets.
Dictionary learning algorithms have been successfully used for both reconstructive and discriminative tasks, where an input signal is represented with a sparse linear combination of dictionary atoms. While these methods are mostly developed for single-modality scenarios, recent studies have demonstrated the advantages …
UniPhyNet improves cognitive load classification accuracy using EEG, ECG, and EDA signals.
problem Classifying cognitive load using multimodal physiological data.
method Unified network architecture integrating multiscale parallel convolutional blocks, ResNet-type blocks, and channel block attention module. Uses bidirectional gated recurrent unit for temporal dependencies.
result Improves raw signal classification accuracy from 70% to 80% (binary) and 62% to 74% (ternary) on CL-Drive dataset.
Modified PCA algorithm with continual learning preserves features of previous modes for multimode process monitoring.
problem Catastrophic forgetting of previous modes in monitoring models for successive modes.
method Modified PCA algorithm with elastic weight consolidation (EWC) to preserve features of previous modes.
result PCA-EWC algorithm effectively monitors multimode processes without performance decrease.
EmbraceNet fusion model for multi-sensor activity recognition.
problem Human activity recognition using multiple sensors is challenging.
method Independent processing of each sensor, feature fusion with EmbraceNet, post-processing, and additional processes.
result Improved performance in SHL recognition challenge.
Proposes a novel network for CTR prediction by learning modality-specific and modality-invariant representations.
problem Learning good representation of items from multimodal features in E-commerce is challenging due to redundant information across modalities.
method Introduces a Multimodal Adversarial Representation Network (MARN) that calculates modality-specific weights and learns modality-invariant representations.
result Consistently achieves remarkable improvements over state-of-the-art methods in CTR prediction.
Deep learning predicts mental disorders from audio and text samples.
problem Predicting mental disorders from speech samples.
method Multimodal deep learning structure using various pre-trained models for audio and text embeddings, transfer learning, and auxiliary corpora.
result Acceptable accuracy in predicting mental disorders through multimodal analysis.
PIMA autoencoders discover shared features in multimodal scientific data.
problem Discovering shared information in high-throughput scientific datasets.
method Physics-informed multimodal autoencoders (PIMA) with Gaussian mixture prior and product of experts formulation.
result Accurate cross-modal inference between images and mechanical stress-strain response in lattice metamaterials.
A new method for projecting multimodal data to a common subspace for one-class classification.
problem Classifying data from multiple sources with varying features.
method Iterative transformation to a common subspace, separate transformations for each modality, regularization strategies.
result Outperforms competing methods across multiple datasets.
Generalized Bilinear Deep Convolutional Neural Networks for Multimodal Biometric Identificationcs.LG
In this paper, we propose to employ a bank of modality-dedicated Convolutional Neural Networks (CNNs), fuse, train, and optimize them together for person classification tasks. A modality-dedicated CNN is used for each modality to extract modality-specific features. We demonstrate that, rather than spatial fusion at the…
The study develops multimodal models to predict 1-year mortality risk from large clinical datasets.
problem Limited datasets in biomedical studies that do not generalize over large heterogeneous datasets.
method Develops multimodal models using a massive clinical dataset of 25 million videos and 2.9 million ECG traces.
result Extremely low-parameter models with optimized feature selection achieve AUC of 0.89.
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) mode…
Benchmarking AutoML for tables with text fields, achieving top performance.
problem Evaluating automated learning systems for tables with text fields.
method Publicly available benchmark with 18 datasets varying in size, types, and feature composition.
result Stack ensembling a multimodal Transformer with various tree models achieved top performance.
MDNE embeds network structures and attributes for better analysis.
problem Preserving both structural and attribute features in network embedding.
method Multimodal Deep Network Embedding (MDNE) using deep model with multiple layers of non-linear functions.
result MDNE outperforms baselines on various tasks with real-world datasets.
A new algorithm learns causal relationships from multimodal data.
problem Discovering causal relationships in exploratory settings without prior information.
method causalPIMA algorithm using multimodal data and physics constraints.
result Learned causal structure and key features in fully unsupervised settings.
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
problem High communication costs in split learning for multimodal models.
method Integrates a compression module that quantizes intermediate features into discrete representations before transmission.
result Achieves an approximate 87.5% reduction in communication overhead with 2-bit quantization.
New method tackles heterogeneous breast cancer imaging data.
problem Challenges in multimodality breast cancer imaging data.
method Developed a multilayer tensor learning method to incorporate heterogeneity.
result Outperforms other existing methods in predicting disease status.
New algorithms improve semantic similarity in mobile datasets by incorporating context.
problem Traditional models fail to capture external context in mobile datasets.
method Developed new algorithms to integrate external features into sentence embeddings and semantic similarity scores.
result Applying PCA with eight components and appending multimodal features improves semantic understanding in tweets.
Deep learning algorithms excel at extracting patterns from raw data, and with large datasets, they have been very successful in computer vision and natural language applications. However, in other domains, large datasets on which to learn representations from may not exist. In this work, we develop a novel multimodal C…
We tackle here the problem of multimodal image non-rigid registration, which is of prime importance in remote sensing and medical imaging. The difficulties encountered by classical registration approaches include feature design and slow optimization by gradient descent. By analyzing these methods, we note the significa…
Simple baselines improve multimodal utterance learning.
problem Learning rich multimodal utterance representations.
method Conditional factorization of utterances into unimodal factors; extending to bimodal and trimodal factors.
result Optimal embeddings can be derived in closed form.
Proposes MR-SNE for multimodal data visualization.
problem Visualizing data from multiple domains with relations across them.
method Extends t-SNE to compute augmented relations and jointly embed them in a low-dimensional space.
result Demonstrates promising performance in visualizing Flickr and Animal with Attributes 2 datasets.
A framework for uncertainty-aware multimodal learning using conformal Shapley intervals.
problem Uncertainty and modality level importance in multimodal learning.
method Introduces conformal Shapley intervals to quantify modality level importance and uncertainty.
result Demonstrates meaningful uncertainty quantification and strong predictive performance.
ECC Analyzer uses LLMs to predict stock volatility from ECCs.
problem Leveraging unstructured ECC data for stock volatility prediction.
method Uses large language models to extract and fuse textual and audio features from ECCs.
result ECC Analyzer outperforms traditional benchmarks in volatility prediction.
This paper addresses the problem of learning dictionaries for multimodal datasets, i.e. datasets collected from multiple data sources. We present an algorithm called multimodal sparse Bayesian dictionary learning (MSBDL). MSBDL leverages information from all available data modalities through a joint sparsity constraint…
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
New model learns multimodal data better than DAGs.
problem Complex multimodal data not well captured by DAGs.
method Latent partial causal model with two latent coupled variables.
result Identifiability result shows representations correspond to latent variables.
TAROT improves data selection for complex multimodal distributions.
problem Suboptimal data selection in multimodal distributions.
method TAROT uses optimal transport theory to select data, addressing greedy heuristics' limitations.
result TAROT outperforms state-of-the-art methods across various deep learning tasks.
MISA combines multiple datasets for better feature extraction.
problem Combining diverse datasets for better feature extraction.
method MISA combines multiple heterogeneous datasets using Kotz distribution and combinatorial optimization.
result MISA produces robust generalization of ICA, IVA, and ISA.
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared …
StoryGen uses images to generate coherent text from multiple images.
problem Generating coherent text from multiple images.
method Designing a Relational Text Data Generator algorithm to relate features from plural images.
result The model can generate meaningful paragraphs containing extracted features from input images.
New method for risk allocation under multimodality of loss distribution.
problem Risk assessment under multimodal conditional loss distribution.
method Maximum Likelihood Allocation (MLA) and multimodality adjustment.
result Multimodality adjustment improves soundness of risk allocations.
One primary focus in multimodal feature extraction is to find the representations of individual modalities that are maximally correlated. As a well-known measure of dependence, the Hirschfeld-Gebelein-Rényi (HGR) maximal correlation becomes an appealing objective because of its operational meaning and desirable propert…
New method captures multimodal disconnectivity in schizophrenia.
problem Misinterpretation of single modality data in schizophrenia research.
method Gaussian graphical model and modularity-based approach on multimodal data.
result Identifies missing links in schizophrenia's default mode network.
Multimodal deep learning improves toxicity prediction accuracy.
problem Improving prediction accuracy of chemical compound toxicity.
method Combining multiple neural network types and data representations.
result Significantly better accuracy on a toxicity benchmark.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
End-to-end CAD system for thyroid nodule classification using multimodal data and expert guidance.
problem Improving accuracy in thyroid nodule classification for clinicians.
method Knowledge-driven DenseNet framework using multimodal ultrasound data and expert cues.
result The proposed system achieves relevant performances in thyroid nodule classification.
InVA models image outcomes from multiple modalities, outperforming standard VAEs.
problem Understanding relationships across multiple imaging modalities in neuroimaging.
method Integrative Variational Autoencoder (InVA) framework for image-on-image regression.
result InVA accurately predicts PET scans from structural MRI, outperforming conventional models.
Model predicts Bitcoin's future movements using multimodal pattern matching.
problem Challenges in predicting Bitcoin's volatile future movements.
method Ranking similar past chart patterns given current chart information.
result Improves directional prediction of Bitcoin's future movements.
DTA aligns multimodal data with prior correspondence knowledge.
problem Challenges in aligning data from different domains.
method Semi-supervised manifold alignment method exploiting prior correspondence knowledge.
result DTA outperforms other methods in aligning multimodal data.
Generative models unify heterogeneous data for multimodal fusion.
problem Learning effective representations of mixed numerical and categorical data.
method Bayesian approach with exponential family distributions and Laplace-Bernstein approximation.
result Generative models enable fusion of multimodal data for various machine learning tasks.
Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Trainingcs.CV
Enhances hand gesture recognition with separate networks and shared features.
problem Improving recognition accuracy of unimodal 3D-CNNs for dynamic hand gestures.
method Separate networks for each modality, collaborative learning, spatiotemporal semantic alignment loss, focal regularization.
result Improves test time recognition accuracy and state-of-the-art performance.
MSB framework improves survival prediction in immunotherapy patients with missing data.
problem High dimensionality and blockwise missingness in multimodal clinical data.
method MSB is a late-fusion framework that independently models modality-specific features before aggregating predictions via cross-validated stacking.
result MSB outperformed baseline algorithms in predicting progression-free survival in lung cancer patients.
Human Activity Recognition (HAR) based on motion sensors has drawn a lot of attention over the last few years, since perceiving the human status enables context-aware applications to adapt their services on users' needs. However, motion sensor fusion and feature extraction have not reached their full potentials, remain…