Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
A framework for uncertainty-aware multimodal learning using conformal Shapley intervals.
problem Uncertainty and modality level importance in multimodal learning.
method Introduces conformal Shapley intervals to quantify modality level importance and uncertainty.
result Demonstrates meaningful uncertainty quantification and strong predictive performance.
This paper investigates uncertainty calibration in multimodal large language models.
problem Challenges in properly calibrating uncertainty in multimodal large language models.
method Investigation of representative MLLMs across various scenarios, including visual fine-tuning and multimodal training.
result MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments.
Unified framework explains few-shot multimodal medical imaging performance.
problem Limited labeled data in rare diseases and low-resource settings.
method PAC learning, VC theory, PAC Bayesian analysis, information gain, Chain of Thought reasoning.
result Unified theoretical framework for few-shot multimodal medical imaging.
MDNs offer a data-efficient alternative to diffusion and flow models for multimodal scientific learning.
problem Capturing multimodal conditional uncertainty in scientific inverse problems.
method Mixture Density Networks (MDNs) as explicit parametric density estimators.
result MDNs achieve superior generalization, interpretability, and sample efficiency in scientific tasks.
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
Deep neural networks (DNNs) provide state-of-the-art results for a multitude of applications, but the approaches using DNNs for multimodal audiovisual applications do not consider predictive uncertainty associated with individual modalities. Bayesian deep learning methods provide principled confidence and quantify pred…
Proposes hinge-Wasserstein to improve uncertainty estimation in regression tasks.
problem Estimating multimodal aleatoric uncertainty in regression tasks from images.
method Regression-by-classification paradigm with hinge-Wasserstein loss.
result Hinge-Wasserstein loss improves uncertainty estimation on challenging tasks.
We propose a novel neural network architecture, named the Global Workspace Network (GWN), which addresses the challenge of dynamic and unspecified uncertainties in multimodal data fusion. Our GWN is a model of attention across modalities and evolving through time, and is inspired by the well-established Global Workspac…
VLM judges rank well but score poorly; task difficulty and annotation quality affect interval width.
problem VLMs as judges lack reliability indicators in multimodal evaluations.
method Conformal prediction using score-token log-probabilities.
result Evaluation uncertainty is task-dependent, affecting interval width and reliability.
Assistive robots can potentially improve the quality of life and personal independence of elderly people by supporting everyday life activities. To guarantee a safe and intuitive interaction between human and robot, human intentions need to be recognized automatically. As humans communicate their intentions multimodall…
FinZero improves financial time series forecasting accuracy with multimodal modeling.
problem Lack of interpretability, uncertainty, and scalability in financial time series forecasting.
method Developed a multimodal pre-trained model FinZero using UARPO method for reasoning, prediction, and uncertainty analysis.
result FinZero achieves an approximate 13.48% improvement in prediction accuracy over GPT-4o in high-confidence group.
Stacking improves inference for multimodal Bayesian posterior distributions.
problem Difficulty of MCMC in moving between modes and underestimation of posterior uncertainty.
method Parallel runs of MCMC, variational, or mode-based inference, combined using Bayesian stacking.
result Stacking efficiently samples from multimodal posterior distributions and represents uncertainty better than variational inference.
Integrates neural encoders into GLMMs for multimodal data analysis.
problem Scalable Bayesian inference for GLMMs assumes low-dimensional tabular predictors and does not handle high-dimensional modalities.
method Jointly learns modality-specific neural encoders with GLMM objective, performs variance-corrected stochastic-gradient MCMC.
result Preserves interpretable fixed and random effects while scaling to large longitudinal datasets.
Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective…
New method uses MRI data to improve PET tomography uncertainty quantification.
problem Improving uncertainty quantification in emission tomography with multimodal data.
method Nonparametric posterior learning technique adapted for Poisson-type data.
result Sampling algorithms are scalable, parallelizable, and easy to implement.
GGMPs improve non-Gaussian conditional density estimation.
problem Multimodality, heteroscedasticity, and strong non-Gaussianity in conditional density estimation.
method GGMP combines local Gaussian mixture fitting, cross-input component alignment, and per-component heteroscedastic GP training.
result GGMPs improve distributional approximation on synthetic and real-world datasets.
Bayesian VFLMSP improves multimodal survival prediction with privacy.
problem Privacy and reliability in multimodal time-to-event prediction.
method Bayesian Vertical Federated Learning (VFL) with differential privacy.
result Consistent improvements in C-index compared to existing methods.
uGMM-NN integrates probabilistic reasoning into neural networks.
problem Capturing multimodality and uncertainty in neural network activations.
method Parameterizes activations as univariate Gaussian mixtures with learnable parameters.
result Competitive discriminative performance with probabilistic activations.
Integrating deep learning with latent state space models has the potential to yield temporal models that are powerful, yet tractable and interpretable. Unfortunately, current models are not designed to handle missing data or multiple data modalities, which are both prevalent in real-world data. In this work, we introdu…
Graph Mixture Density Networks model multimodal data on graphs.
problem Challenging conditional density estimation problems with structured data.
method Combining mixture models and graph representation learning.
result Significant improvement in likelihood of epidemic outcomes.
MC Dropout is re-evaluated as not Bayesian, affecting predictive posterior and multimodality.
problem MC Dropout's Bayesian properties and predictive posterior are questioned.
method Re-evaluation of MC Dropout's properties, including a new VI engine in pytorch.
result MC Dropout does not produce a faithful Bayesian predictive posterior.
Bayesian neural networks reveal multimodal predictive distributions.
problem Uncertainty quantification and interpretability in neural networks.
method Discretized prior for inner layer weights, Gaussian mixture approximation of posterior predictive distribution.
result Distinct parameter realizations can produce the same training error but different posterior predictive distributions.
Hybrid method improves sampling from multimodal distributions.
problem Sampling from multimodal posterior distributions efficiently.
method Jump-Diffusion Langevin Dynamics hybrid with Metropolis.
result Calibrated hybrid method outperforms pure methods.
New method optimizes portfolios by dynamically integrating ESG constraints.
problem Static ESG scores mismatch sequential portfolio decisions.
method MACF-X, a family of adapters that learns ESG costs from multimodal evidence.
result Reduces tail ESG budget pressure while maintaining financial performance.
Unified model improves object-centric world modeling with new abilities.
problem Integration of recent advances and temporal imagination abilities.
method Generative Structured World Models (G-SWM) framework.
result G-SWM achieves best or comparable performance in temporal generation.
Study creates a multimodal learning framework for CVD risk prediction.
problem Predicting cardiovascular disease risk in diverse populations.
method Combines cross modal transformers, graph neural networks, and causal representation learning.
result Model predicts personalized CVD risk with causal invariance across subpopulations.
Nested Slice Sampling accelerates Nested Sampling for GPU acceleration.
problem Challenging inference for complex, multimodal targets.
method Vectorized Nested Slice Sampling using Hit-and-Run Slice Sampling.
result NSS maintains accurate evidence estimates and high-quality posterior samples, robust on multimodal problems.
Novel SVAE learns interpretable discrete data representations from deep learning.
problem Learning interpretable discrete data representations from deep learning.
method Structured variational autoencoder (SVAE) with novel optimization algorithms.
result First competitive comparisons with state-of-the-art time series models.
We introduce a Bayesian framework for inference with a supervised version of the Gaussian process latent variable model. The framework overcomes the high correlations between latent variables and hyperparameters by using an unbiased pseudo estimate for the marginal likelihood that approximately integrates over the late…
This paper applies quantum probability theory to model asset returns, avoiding assumptions about quantum effects.
problem Modeling asset returns with classical probability theory.
method Derives a Schrödinger-like trading equation using quantum probability, linking it to traders' decisions and market behaviors.
result Quantum probability can describe multimodal distributions of asset returns without assuming quantum effects.
DGMEs use Gaussian mixtures to quantify uncertainty in deep learning.
problem Quantifying uncertainty in complex predictive densities.
method DGMEs use a Gaussian mixture model with an EM algorithm for parameter learning.
result DGMEs outperform state-of-the-art models in uncertainty quantification.
Due to the need for robust uncertainty quantification, Bayesian neural learning has gained attention in the era of deep learning and big data. Markov Chain Monte-Carlo (MCMC) methods typically implement Bayesian inference which faces several challenges given a large number of parameters, complex and multimodal posterio…
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of j…
CNR uses convex optimization to estimate conditional distributions.
problem Estimating uncertainty in predictions and posterior conditional distributions.
method Convex optimization of a posterior defined via non-linear transformations on Gaussians.
result CNR can fit arbitrary conditional distributions, including multimodal and non-symmetric ones.
Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
Multimodal fusion is considered a key step in multimodal tasks such as sentiment analysis, emotion detection, question answering, and others. Most of the recent work on multimodal fusion does not guarantee the fidelity of the multimodal representation with respect to the unimodal representations. In this paper, we prop…
In recent studies on model-based reinforcement learning (MBRL), incorporating uncertainty in forward dynamics is a state-of-the-art strategy to enhance learning performance, making MBRLs competitive to cutting-edge model free methods, especially in simulated robotics tasks. Probabilistic ensembles with trajectory sampl…
Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exp…
NE-GMM uses ES and GMM to improve uncertainty quantification.
problem Challenges in estimating mean and variance of complex distributions.
method Integrates Gaussian Mixture Model with Energy Score.
result NE-GMM outperforms in predictive accuracy and uncertainty quantification.
Develops a contrastive framework for data-efficient multimodal learning.
problem Expensive training of multimodal generative models requiring related multimodal data.
method Contrastive framework for multimodal learning, distinguishing related from unrelated data.
result Data-efficient multimodal learning on challenging datasets for various VAE models.
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) mode…
Multimodalities provide promising performance than unimodality in most tasks. However, learning the semantic of the representations from multimodalities efficiently is extremely challenging. To tackle this, we propose the Transformer based Cross-modal Translator (TCT) to learn unimodal sequence representations by trans…
Novel framework uses synthetic data to quantify uncertainty in complex data.
problem Uncertainty quantification in complex, unstructured data.
method Perturbation-Assisted Sample Synthesis (PASS) and Perturbation-Assisted Inference (PAI) framework.
result Statistically guaranteed validity in inference, enhancing reliability of synthetic data.
Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent r…
Paper proposes multimodal contrastive learning for EHR data.
problem Separate treatment of structured and unstructured EHR data.
method Proposes a multimodal feature embedding generative model and a multimodal contrastive loss.
result Multimodal learning yields better feature representation than single-modality learning.
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised mult…