Multimodal bitransformer boosts image-text classification.
problem Combining text and image modalities for improved classification.
method Supervised multimodal bitransformer model integrating text and image encoders.
result State-of-the-art performance on multimodal classification benchmarks.
TCT learns multimodal sequence representations by translating from related sequences.
problem Challenges in learning semantic representations from multimodalities.
method Transformer based Cross-modal Translator (TCT) combined with Multimodal Transformer Network (MTN).
result Proposed method achieves new state-of-the-art performance on video-grounded dialogue.
New ELBO formulation improves multimodal learning.
problem Existing ELBO models struggle with multimodal data.
method Proposes a generalized ELBO for multimodal data.
result Demonstrates improved performance over state-of-the-art models.
Proposes a multimodal deep generative model for semi-supervised learning with class imbalance.
problem Class imbalance in semi-supervised learning with partial supervision.
method Separate encoders for each modality, sharing latent variables, and using Student's t-distributions for prior, encoder, and decoder.
result Outperforms baseline methods in generalization and classification performance for partially labeled multimodal data.
Study quantifies interactions between unlabeled multimodal data.
problem Understanding how modalities combine in semi-supervised settings.
method Information-theoretic definitions and bounds derivation.
result Validated lower and upper bounds accurately track true interactions.
DAIF learns fusion structure from data to improve multimodal supervised learning.
problem Tackles the challenge of determining optimal fusion granularity across heterogeneous data sources.
method Combines random matrix theory and non-parametric dependence measures to learn fusion structure directly from data.
result DAIF outperforms state-of-the-art techniques in predicting T-cell differentiation marker protein expression and patient survival.
Simple baselines improve multimodal utterance learning.
problem Learning rich multimodal utterance representations.
method Conditional factorization of utterances into unimodal factors; extending to bimodal and trimodal factors.
result Optimal embeddings can be derived in closed form.
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
MSBDL learns multimodal dictionaries with flexibility and superior performance.
problem Learning dictionaries for datasets from multiple data sources.
method Multimodal sparse Bayesian dictionary learning (MSBDL) with joint sparsity constraint.
result MSBDL outperforms existing methods on synthetic and real datasets.
Unified approach for multimodal data prediction using synthetic data generation.
problem Challenges in integrating heterogeneous data types for accurate predictive performance.
method Generative Distribution Prediction (GDP) framework that uses multimodal synthetic data generation.
result Empirical validation across four tasks demonstrates versatility and effectiveness of GDP.
Study presents a dataset and evaluation framework for representation learning in complex multimodal systems.
problem Lack of large-scale standard datasets for representation learning in complex multimodal systems.
method Implemented and compared several approaches to representation learning on a large-scale dataset for landing an airplane.
result Representations can be used for various applications including anomaly detection and optimal control.
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
A new model learns from multiple types of data without needing complete information.
problem Learning from multiple types of data without complete information.
method Multimodal variational autoencoder (MVAE) with product-of-experts inference network and sub-sampled training.
result Matches state-of-the-art performance with fewer parameters and is robust to incomplete supervision.
daep learns from irregular, multimodal astronomical data.
problem Learning from irregular, multimodal astronomical sequences.
method Diffusion Autoencoder with Perceivers (daep) tokenizes, compresses, and reconstructs data.
result daep outperforms VAE and maep baselines in reconstruction and fine-scale structure preservation.
Improved multimodal variational models capture more complex joint distributions.
problem Limited expressiveness of multimodal variational models.
method Used normalizing flows to approximate and transform a simple parametric joint posterior into a more complex one.
result The model improves on state-of-the-art multimodal variational methods on various tasks.
Open-FinLLMs tackle financial tasks with multimodal capabilities.
problem Financial LLMs lack multimodal capabilities and real-world applicability.
method Developed Open-FinLLMs, an open-source multimodal financial LLM suite.
result Open-FinLLMs outperform advanced financial and general LLMs in diverse tasks.
PVAE learns disentangled representations from multimodal data.
problem Learning disentangled representations from multimodal sensory data.
method Partitioned Variational Autoencoder (PVAE) with multimodal generative model and training objectives.
result PVAE achieves over 99% accuracy on both modalities for semantic units.
The paper introduces multimodal generative models to improve data marginal likelihood.
problem Improving data marginal likelihood in multimodal settings.
method Derives variational bounds on the evidence for multimodal deep generative models, generalizes objectives for different model types, and benchmarks across various datasets.
result Multimodal VAEs excel in image, label, and text datasets with and without weak supervision.
New method handles missing data and multiple data types in time series models.
problem Handling missing data and multiple data modalities in time series models.
method Factorized inference method for Multimodal Deep Markov Models (MDMMs).
result Method performs well even with high levels of missing data and outperforms existing approaches.
Bayesian approach for handling incomplete clinical data.
problem Challenges in machine learning with multimodal, incomplete clinical data.
method Generative and discriminative learning, semi-supervised strategy, imputation of missing views.
result Automatic imputation of missing views and robust inference across different data sources.
Benchmarking AutoML for tables with text fields, achieving top performance.
problem Evaluating automated learning systems for tables with text fields.
method Publicly available benchmark with 18 datasets varying in size, types, and feature composition.
result Stack ensembling a multimodal Transformer with various tree models achieved top performance.
Paper proposes C-STM for multimodal neuroimaging data classification.
problem Multimodal neuroimaging data fusion for better classification.
method Coupled Support Tensor Machine (C-STM) using latent factors from ACMTF.
result C-STM achieves better classification performance than single-mode classifiers.
A new framework extracts informative features from multiple data types.
problem Finding maximally correlated feature representations from multimodal data.
method Soft-HGR framework, which relaxes whitening constraints while preserving feature geometry.
result Soft-HGR learns more informative feature mappings and is more efficient to optimize.
A new algorithm learns causal relationships from multimodal data.
problem Discovering causal relationships in exploratory settings without prior information.
method causalPIMA algorithm using multimodal data and physics constraints.
result Learned causal structure and key features in fully unsupervised settings.
A new parallel MCMC algorithm improves topic modeling without communication.
problem Quasi-ergodicity problem in topic modeling due to multimodal topic distributions.
method Developed an embarrassingly parallel MCMC algorithm for sLDA by switching topic combination and labeling prediction.
result Out-of-sample prediction performance is comparable to non-parallel sLDA but computation time is significantly reduced.
Hybrid model for multimodal distributions using diffusion and classification.
problem Sampling from multimodal distributions with correct proportions.
method Divide-and-conquer strategy: identify modes, train classifiers, diffusion models, bridge sampling.
result Framework effectively handles multimodal distributions in high dimensions.
DTA aligns multimodal data with prior correspondence knowledge.
problem Challenges in aligning data from different domains.
method Semi-supervised manifold alignment method exploiting prior correspondence knowledge.
result DTA outperforms other methods in aligning multimodal data.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
RandomNet uses random search to design neural architectures without much human intervention.
problem Designing neural architectures without excessive human intervention.
method Random search strategy for multimodal neural architecture design.
result RandomNet performs close to state-of-the-art on AV-MNIST with minimal human supervision.
Data balancing reduces variance in machine learning models.
problem Reduction of variance in machine learning models.
method Non-asymptotic statistical bound and eigenvalue decay of Markov operators.
result Data balancing across modalities and sources reduces variance.
Framework translates images between domains without supervision.
problem Challenges in unsupervised image-to-image translation, especially handling multimodality.
method Proposes a Multimodal Unsupervised Image-to-Image Translation (MUNIT) framework, decomposing images into content and style codes.
result Demonstrates improved generation of diverse outputs from a single source image.
Bayesian approach to data association using Gaussian processes.
problem Separating data from different generating processes.
method Fully Bayesian approach with Gaussian process priors for structure encoding and doubly stochastic variational inference.
result Efficient learning scheme for deep Gaussian process priors.
Paper proposes Seq2Seq models for multimodal sentiment analysis.
problem Learning representations from multiple modalities in machine learning.
method Two unsupervised Seq2Seq models for multimodal sentiment analysis.
result Seq2Seq models improve F1 Score by twelve points in Bimodal sentiment analysis.
Integrates neural encoders into GLMMs for multimodal data analysis.
problem Scalable Bayesian inference for GLMMs assumes low-dimensional tabular predictors and does not handle high-dimensional modalities.
method Jointly learns modality-specific neural encoders with GLMM objective, performs variance-corrected stochastic-gradient MCMC.
result Preserves interpretable fixed and random effects while scaling to large longitudinal datasets.
New method improves self-supervised representation learning using probabilistic modeling and Monte Carlo integration.
problem Improving self-supervised representation learning for multimodal data.
method Discriminative probabilistic modeling with multiple importance sampling (MIS) for robust Monte Carlo integration.
result Proposes a novel non-parametric method for approximating conditional probability densities through convex optimization.
This paper proposes a model to learn multimodal representations robust to missing data.
problem Learning multimodal representations from heterogeneous sources of information.
method Optimizes a joint generative-discriminative objective across multimodal data and labels, factorizing representations into multimodal discriminative and modality-specific generative factors.
result The proposed model achieves state-of-the-art performance on six multimodal datasets and can reconstruct missing modalities without significant performance drop.
STE algorithm predicts human behavior from sensor data.
problem Predicting human behavior from noisy, incomplete, and heterogeneous bio-behavioral data.
method Supervised Tensor Embedding (STE) algorithm for high-dimensional multimodal data.
result Features selection improves prediction performance.
Develops a contrastive framework for data-efficient multimodal learning.
problem Expensive training of multimodal generative models requiring related multimodal data.
method Contrastive framework for multimodal learning, distinguishing related from unrelated data.
result Data-efficient multimodal learning on challenging datasets for various VAE models.
This paper explores how the latent structure affects clustering in GAN-generated data.
problem Achieving well-clustered data in GAN-generated spaces, especially with class imbalance.
method Derives conditions for faithful clustering in GANs, including multimodal latent space, latent space inversion, and cluster priors imposition.
result Necessary and sufficient conditions for clustering in GANs are identified, and their importance is demonstrated through ablative studies.
New unsupervised learning framework for sound recognition.
problem Learning sound recognition without explicit labels.
method Combines self-supervised and clustering objectives with active learning.
result Achieves state-of-the-art unsupervised audio representation with reduced labels.
FMT model improves multimodal sequential learning across language, vision, and acoustic data.
problem Modeling spatio-temporal dynamics across multiple modalities.
method Factorized Multimodal Transformer (FMT) that models intramodal and intermodal dynamics in a factorized manner.
result FMT outperforms existing models on 3 datasets and 21 labels, setting new state of the art.
GROOVE learns representations for weakly paired multimodal data.
problem Learning representations for high-content perturbation data with weakly paired samples.
method GroupCLIP contrastive loss integrated with an autoencoder framework.
result GROOVE performs on par with or outperforms existing approaches for cross-modal tasks.
Study develops a semi-supervised deep ResNet for Wi-Fi mode detection.
problem Utilizing Wi-Fi signals for multimodal transportation mode detection with limited labeled data.
method Semi-supervised deep residual network (ResNet) framework.
result Framework achieves high prediction accuracy (81.8% for walking, 82.5% for biking, 86.0% for driving).
Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
This paper strengthens the computational separation between multimodal and unimodal learning, showing unimodal learning is hard on typical instances.
problem Theoretical justification for empirical success of multimodal machine learning.
method Introduced a stronger average-case computational separation between unimodal and multimodal learning.
result For typical instances, unimodal learning is computationally hard, while multimodal learning is easy.
Paper learns multimodal transition dynamics for RL using conditional VI.
problem Learning stochastic, multimodal transition dynamics in RL.
method Conditional variational inference (VI) for complex stochasticity.
result VI successfully predicts multimodal outcomes but ignores deterministic parts.
We improve conditional VAEs by incentivizing informative latent variables.
problem Structured-prediction tasks with one-to-many mappings.
method Modify latent variable model and introduce a multimodal prior.
result Significantly higher generalisation capability demonstrated on various datasets.
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.