Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent r…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “multimode processes”
GGMPs improve non-Gaussian conditional density estimation.
problem Multimodality, heteroscedasticity, and strong non-Gaussianity in conditional density estimation.
method GGMP combines local Gaussian mixture fitting, cross-input component alignment, and per-component heteroscedastic GP training.
result GGMPs improve distributional approximation on synthetic and real-world datasets.
Modified PCA algorithm with continual learning preserves features of previous modes for multimode process monitoring.
problem Catastrophic forgetting of previous modes in monitoring models for successive modes.
method Modified PCA algorithm with elastic weight consolidation (EWC) to preserve features of previous modes.
result PCA-EWC algorithm effectively monitors multimode processes without performance decrease.
Study improves document processing in banking with multimodal analytics.
problem Raising operational efficiency in banking through document-intensive processes.
method Comparative analysis of text classifiers and multimodal model (LayoutXLM) on company register extracts.
result Incorporating layout information in a model substantially increases performance.
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of j…
New results show contrastive learning can recover shared factors in multimodal data.
problem Understanding when contrastive learning can recover shared latent factors in multimodal data.
method New identifiability results for multimodal contrastive learning, distinguishing between multi-view and multimodal settings.
result Contrastive learning can block-identify shared latent factors in multimodal data, even with dependencies.
Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs)…
Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exp…
Integrating visual and linguistic information into a single multimodal representation is an unsolved problem with wide-reaching applications to both natural language processing and computer vision. In this paper, we present a simple method to build multimodal representations by learning a language-to-vision mapping and…
EmbraceNet fusion model for multi-sensor activity recognition.
problem Human activity recognition using multiple sensors is challenging.
method Independent processing of each sensor, feature fusion with EmbraceNet, post-processing, and additional processes.
result Improved performance in SHL recognition challenge.
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
Spectral Diffusion Processesstat.ML
Extends SGM to functional spaces for multimodal data.
problem Modeling densities in functional spaces.
method Represent data in spectral space, dissociate stochastic and space-time components, use SGM for sampling.
result Demonstrates effectiveness on multimodal datasets.
This research improves multimodal systems by adding a second objective and regularisation methods.
problem Improving performance of multimodal systems with multiple objectives and regularisation.
method Introduces a second objective over multimodal fusion using variational inference and regularisation methods.
result Demonstrates potential for multiple objectives and probabilistic methods to lower variance and improve generalisation.
CQNPs enhance predictive performance and distribution modeling using quantile regression.
problem Limited predictive likelihood of Gaussian models for complex distributions.
method Introducing Conditional Quantile Neural Processes (CQNPs) that focus on estimating informative quantiles.
result Significant improvements in predictive performance and better modeling of multimodal distributions.
New method uses birth-death process and exploration component to accelerate sampling from multimodal distributions.
problem Sampling from multimodal probability distributions efficiently.
method Combines birth-death process and exploration component to accelerate sampling.
result Proves exponential asymptotic convergence under mild assumptions.
Framework detects anomalies in industrial processes using deep learning.
problem Detect anomalies in complex industrial processes.
method Causal-based framework with unsupervised deep learning.
result Successfully validated abstract contexts of blast furnace assets.
We study the problem of multimodal generative modelling of images based on generative adversarial networks (GANs). Despite the success of existing methods, they often ignore the underlying structure of vision data or its multimodal generation characteristics. To address this problem, we introduce the Dirichlet prior fo…
There has been an increased interest in multimodal language processing including multimodal dialog, question answering, sentiment analysis, and speech recognition. However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption. To address these c…
Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriorsstat.ME
Stacking improves inference for multimodal Bayesian posterior distributions.
problem Difficulty of MCMC in moving between modes and underestimation of posterior uncertainty.
method Parallel runs of MCMC, variational, or mode-based inference, combined using Bayesian stacking.
result Stacking efficiently samples from multimodal posterior distributions and represents uncertainty better than variational inference.
A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistq-fin.TR
FinAgent tackles financial trading with multimodal data and advanced AI.
problem Challenges in handling multimodal financial data and limited generalizability.
method Multimodal foundational agent with tool augmentation, dual-level reflection, and diversified memory retrieval.
result Significantly outperforms state-of-the-art baselines in financial trading tasks.
Study improves stock movement prediction using multimodal data.
problem Inaccurate stock movement prediction due to incomplete multimodal data integration.
method Introduces MSGCA framework for robust multimodal fusion.
result MSGCA framework outperforms existing methods by 21.7% on multimodal datasets.
Multimodal ML models can process data in multiple modalities (e.g., video, images, audio, text) and are useful for video content analysis in a variety of problems (e.g., object detection, scene understanding). In this paper, we focus on the problem of video categorization by using a multimodal approach. We have develop…
New framework estimates graph from multimodal functional data.
problem Estimating graph from joint multimodal functional data.
method Integrative framework using partial correlation operator.
result Estimator converges to stationary point with quantifiable error.
PIMA autoencoders discover shared features in multimodal scientific data.
problem Discovering shared information in high-throughput scientific datasets.
method Physics-informed multimodal autoencoders (PIMA) with Gaussian mixture prior and product of experts formulation.
result Accurate cross-modal inference between images and mechanical stress-strain response in lattice metamaterials.
lamBERT learns language and actions using multimodal BERT.
problem Learning language and actions in complex environments.
method Extending BERT to multimodal representation and integrating with reinforcement learning.
result lamBERT model achieved higher rewards in multitask and transfer settings.
This study improves stock price prediction using multimodal data.
problem Improving financial asset price forecasting accuracy.
method Combining candlestick time series and textual news flow data using LSTM and pre-trained models.
result Textual modality reduces MAPE by 55%.
Current high-throughput data acquisition technologies probe dynamical systems with different imaging modalities, generating massive data sets at different spatial and temporal resolutions posing challenging problems in multimodal data fusion. A case in point is the attempt to parse out the brain structures and networks…
New model learns multimodal data better than DAGs.
problem Complex multimodal data not well captured by DAGs.
method Latent partial causal model with two latent coupled variables.
result Identifiability result shows representations correspond to latent variables.
UniPhyNet improves cognitive load classification accuracy using EEG, ECG, and EDA signals.
problem Classifying cognitive load using multimodal physiological data.
method Unified network architecture integrating multiscale parallel convolutional blocks, ResNet-type blocks, and channel block attention module. Uses bidirectional gated recurrent unit for temporal dependencies.
result Improves raw signal classification accuracy from 70% to 80% (binary) and 62% to 74% (ternary) on CL-Drive dataset.
DTA aligns multimodal data with prior correspondence knowledge.
problem Challenges in aligning data from different domains.
method Semi-supervised manifold alignment method exploiting prior correspondence knowledge.
result DTA outperforms other methods in aligning multimodal data.
Graph-based multimodal federated learning for HAR improves accuracy and privacy.
problem Challenges in HAR due to noisy data, incomplete measurements, and privacy concerns.
method Proposes GraMFedDHAR, a Graph-based Multimodal Federated Learning framework for HAR tasks, using modality-specific graphs, residual GCNs, and attention-based fusion.
result Experimental results show up to 13 percent performance improvement for MultiModalGCN under differential privacy constraints.
DAM: A Universal Dual Attention Mechanism for Multimodal Timeseries Cryptocurrency Trend Forecastingecon.GN
DAM improves cryptocurrency trend forecasting using multimodal data.
problem Simplistic merging of sentiment data in cryptocurrency trend forecasting.
method Dual Attention Mechanism (DAM) integrating financial metrics and sentiment analysis.
result DAM outperforms conventional models by up to 20% in prediction accuracy.
A new model DKMPP integrates covariates and uses an integration-free method for spatio-temporal point processes.
problem Training intractable deep spatio-temporal point processes with multimodal covariates.
method DKMPP uses a deep kernel to model complex relationships and an integration-free score matching method.
result DKMPP and score-based estimators outperform baseline models in spatio-temporal point processes.
RiskLabs uses LLMs to predict financial risks from multimodal data.
problem Financial risk prediction using AI techniques.
method Integrates multimodal financial data (textual, vocal, time series, news) into LLMs for prediction.
result Empirical results show effectiveness in forecasting market volatility and variance.
Enhances stock movement prediction using Higher Order Transformers for multimodal time-series data.
problem Predicting stock movements in financial markets with complex dynamics.
method Introduced Higher Order Transformers, extending self-attention and transformer architecture to capture complex market dynamics. Employed low-rank tensor decomposition and kernel attention to manage computational complexity. Integrated technical and fundamental analysis from historical prices and tweets.
result Demonstrated effectiveness of the method on the Stocknet dataset, improving stock movement prediction.
Model predicts Bitcoin's future movements using multimodal pattern matching.
problem Challenges in predicting Bitcoin's volatile future movements.
method Ranking similar past chart patterns given current chart information.
result Improves directional prediction of Bitcoin's future movements.
ICYM2I corrects missingness bias in multimodal learning.
problem Missingness patterns between source and target environments affect multimodal learning performance.
method ICYM2I uses inverse probability weighting to correct missingness bias in predictive performance and information gain.
result ICYM2I improves multimodal learning performance by accounting for missingness.
MAESTRO improves multimodal learning for dynamic time series with adaptive attention and robustness.
problem Challenges in multimodal learning, especially in healthcare and daily living.
method Dynamic intra- and cross-modal interactions, symbolic tokenization, adaptive attention budgeting, sparse cross-modal attention, MoE mechanism.
result Average relative improvements of 4% and 8% over existing multimodal and multivariate approaches, respectively, under complete observations.
Deep Gaussian Processes improve likelihood-free inference for complex distributions.
problem Limited flexibility of Bayesian Optimization with GPs for multimodal distributions.
method Proposes Deep Gaussian Processes (DGPs) as a surrogate model for likelihood-free inference.
result DGPs outperform GPs on multimodal distributions while maintaining comparable performance on unimodal cases.
Unified model predicts stock and systemic risks from diverse financial data.
problem Isolating financial tasks leads to missed cross-scale dependencies.
method Shared Transformer backbone with modular task heads for cross-modal attention and multi-task optimization.
result Uni-FinLLM significantly outperforms baselines in stock forecasting, credit-risk assessment, and systemic-risk detection.
Neural network approximates diffusion bridges for efficiency and robustness.
problem Efficient simulation of conditioned diffusion processes, especially rare events and multimodal distributions.
method Trains a neural network to approximate bridge dynamics, eliminating MCMC and score modeling.
result Efficient sampling of conditioned diffusion bridges at comparable cost to unconditioned process.
Survey of multimodal deep generative models for diverse data types.
problem Inference of shared representations and cross-modal generation from heterogeneous multimodal data.
method Variational autoencoders and other deep generative models.
result A comprehensive survey of multimodal deep generative models.
Multimodal fusion is considered a key step in multimodal tasks such as sentiment analysis, emotion detection, question answering, and others. Most of the recent work on multimodal fusion does not guarantee the fidelity of the multimodal representation with respect to the unimodal representations. In this paper, we prop…
Data is one of the essential ingredients to power deep learning research. Small datasets, especially specific to medical institutes, bring challenges to deep learning training stage. This work aims to develop a practical deep multimodal that can classify patients into abnormal and normal categories accurately as well a…
This work extends ME-RL using diffusion models to sample optimal policies.
problem Sampling from the optimal policy trajectory distribution in ME-RL.
method Introducing Diffusion-Augmented Markov Decision Processes (DA-MDPs) to minimize reverse KL divergence.
result DA-MDPs enable seamless integration into various ME-RL methods and outperform baselines.
Develops a contrastive framework for data-efficient multimodal learning.
problem Expensive training of multimodal generative models requiring related multimodal data.
method Contrastive framework for multimodal learning, distinguishing related from unrelated data.
result Data-efficient multimodal learning on challenging datasets for various VAE models.
The paper uses spectral flow on SPD matrices to analyze multimodal data.
problem Analyzing data from multiple sensors with shared and unique sources.
method Combines manifold learning with Riemannian geometry of SPD matrices.
result Spectral analysis of kernels on SPD manifold reveals common and unique components.
New model improves multimodal autoencoders by learning joint and conditional distributions.
problem Limitations in recent multimodal autoencoders restrict their quality on complex datasets.
method Proposes a multistage training process with variational inference and Normalizing Flows, leveraging shared modality information.
result Achieves state-of-the-art results on benchmark datasets.