The paper analyzes and proposes an algorithm for multi-modal nonlinear embeddings with theoretical performance bounds.
problem Generalizability of multi-modal nonlinear embeddings to unseen data.
method Theoretical analysis and a multi-modal nonlinear representation learning algorithm motivated by performance bounds.
result The proposed algorithm yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
Contrastive learning adapts to data intrinsic dimensions, learning low-dimensional representations.
problem Learning high-dimensional representations from multi-modal data.
method Multi-modal contrastive learning with temperature optimization.
result Contrastive learning adapts to intrinsic dimensions of data, not specified dimensions.
METEOR learns efficient representations from multi-modal data streams.
problem Efficiently interpreting multi-modal information in complex environments.
method METEOR learns compact representations by sharing parameters within semantically meaningful groups and preserving domain-agnostic semantics.
result METEOR reduces memory usage by around 80% compared to conventional methods.
New framework shows cross-attention improves multi-modal in-context learning.
problem Understanding multi-modal in-context learning in neural networks.
method Mathematical framework and linearized cross-attention mechanism.
result Cross-attention mechanism is provably optimal for multi-modal in-context learning.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
PyTorch Frame simplifies multi-modal tabular learning with modular data and model handling.
problem Handling complex multi-modal tabular data in deep learning.
method A PyTorch-based framework that provides a data structure, model abstraction, and integration with external models.
result Demonstrated the effectiveness of PyTorch Frame in implementing and applying diverse tabular models to complex multi-modal tabular data.
Adaptive anchor methods improve multi-modal learning by balancing intra-modal and inter-modal information.
problem Fixed anchor methods limit multi-modal learning by over-reliance on a single modality and inadequate cross-modal correlation.
method Adaptive anchor methods using centroid-based anchors from all modalities.
result Adaptive anchor methods like CentroBind consistently outperform fixed anchor methods across various datasets.
Develops multi-modal neural network models for improved prediction and uncertainty quantification.
problem Improving prediction accuracy and uncertainty quantification for multi-modal data.
method Multi-modal Bayesian neural network models with conjugate last-layer estimation using SVI.
result Improved prediction accuracy and uncertainty quantification compared to uni-modal models.
This paper reviews deep learning for multi-modality medical image segmentation.
problem Improving segmentation accuracy in medical images using multiple modalities.
method Overview of deep learning and multi-modal medical image segmentation, analysis of different network architectures and fusion strategies.
result Later fusion of modalities can lead to more accurate segmentation results.
This paper learns multi-modal embeddings from text, audio, and video views/modes of data in order to improve upon down-stream sentiment classification. The experimental framework also allows investigation of the relative contributions of the individual views in the final multi-modal embedding. Individual features deriv…
Framework for handling long-tailed multi-modal data.
problem Class imbalance and long-tailed distributions in multi-modal data.
method Multi-expert architecture with modality-specific networks and dynamic fusion weights.
result Framework outperforms existing methods in long-tailed, class-imbalanced scenarios.
This work improves multi-modal generative models by using permutation-invariant neural networks.
problem Improving multi-modal generative models with tighter variational objectives.
method Developed more flexible aggregation schemes based on permutation-invariant neural networks.
result Our variational objective and flexible aggregation models can better approximate the true joint distribution.
This study optimizes multi-modal learning thresholds and algorithms in high dimensions.
problem Optimizing multi-modal learning performance in high-dimensional data.
method Analytical quantification and derivation of AMP algorithm with state evolution analysis.
result Bayes-optimal performance and recovery thresholds derived for multi-modal data.
Study reveals efficient recovery of multi-modal signals via Bayesian methods and sequential learning.
problem Recovering multiple high-dimensional signals from correlated modalities.
method Bayesian Approximate Message Passing and Sequential Curriculum Learning.
result Sequential learning strategy optimally recovers weak signals in multi-modal settings.
AF improves sampling from high-dimensional, multi-modal distributions.
problem Sampling from high-dimensional, multi-modal distributions is challenging.
method Annealing Flow (AF) using Continuous Normalizing Flow (CNF) with dynamic Optimal Transport (OT) objective and annealing procedures.
result AF significantly improves training efficiency and stability, outperforming state-of-the-art methods.
Human behavior expression and experience are inherently multi-modal, and characterized by vast individual and contextual heterogeneity. To achieve meaningful human-computer and human-robot interactions, multi-modal models of the users states (e.g., engagement) are therefore needed. Most of the existing works that try t…
Paper introduces topological eigenvalue theorems for tensor analysis in multi-modal data.
problem Lack of deep understanding of tensor structures in multi-modal data fusion.
method Introduces topological perspective to tensor eigenvalue analysis, linking eigenvalues to topological features.
result Establishes new theorems that enhance understanding of tensor structures in data fusion.
This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answeri…
A new algorithm flattens multi-modal distributions for better deep learning.
problem Bayesian learning in big data with multi-modal distributions.
method Contour Stochastic Gradient Langevin Dynamics (CSGLD) algorithm.
result The CSGLD algorithm avoids local traps in deep neural networks.
Triple-GAIL learns from multiple sources to improve imitation learning for complex behaviors.
problem Limited scalability of GAIL in real-world scenarios like autonomous vehicles.
method Integrates expert demonstrations and generated experiences with an auxiliary skill selector.
result Triple-GAIL outperforms state-of-the-art methods in learning complex behaviors.
Method predicts NBA players' multi-modal movement trajectories.
problem Understanding NBA players' decision-making during games.
method LSTM-based architecture with multi-modal loss function.
result Method outperforms state-of-the-art in predicting realistic trajectories.
Clinical diagnostic decision making and population-based studies often rely on multi-modal data which is noisy and incomplete. Recently, several works proposed geometric deep learning approaches to solve disease classification, by modeling patients as nodes in a graph, along with graph signal processing of multi-modal …
This work improves manifold learning for multi-modal data.
problem Distortions and modeling errors in multi-modal data.
method Isometrizing learned Riemannian structure and balancing regularity and expressivity.
result The synergy of proposed approaches enhances manifold learning.
Diffusion models learn multi-modal distributions with optimal efficiency.
problem Learning high-dimensional distributions with low-dimensional multi-modal structures.
method Score-based diffusion models, focusing on subgaussian distributions within subspaces.
result Diffusion models require O ~ ( ε − k ∨ 2 ) \widetilde{O}(\varepsilon^{-k \vee 2}) O ( ε − k ∨ 2 ) samples for 1-Wasserstein ε \varepsilon ε error, improving over prior guarantees. Framework for causal discovery using multi-modal data.
problem Failure of representation learning in causal tasks.
method Statistical and computational framework combining representation learning and causal inference.
result Effective use of observational and perturbational data for causal discovery.
New divergences improve score-based methods for multi-modal distributions.
problem Blindness problem in score-based divergences for multi-modal distributions.
method Proposed a new family of divergences to mitigate blindness.
result Improved performance in density estimation compared to traditional approaches.
Recent work on imitation learning has generated policies that reproduce expert behavior from multi-modal data. However, past approaches have focused only on recreating a small number of distinct, expert maneuvers, or have relied on supervised learning techniques that produce unstable policies. This work extends InfoGAI…
Paper analyzes deep learning models for credit rating prediction using text and numerical data.
problem Improving credit rating prediction using multi-modal deep learning.
method Testing different deep learning models and fusion strategies for structured and unstructured datasets.
result CNN-based multi-modal model with two fusion strategies outperformed other models.
A new memory-based fusion layer improves multi-modal deep learning performance.
problem Improving performance of multi-modal deep learning by addressing long-term dependencies.
method Introducing a Memory based Attentive Fusion (MBAF) layer that incorporates both current and long-term dependencies.
result The MBAF layer enhances fusion and improves performance across different modalities and networks.
ROME improves density estimation for multi-modal, non-normal data.
problem Robust multi-modal density estimation in non-normal, highly correlated distributions.
method ROME uses clustering to segment multi-modal data into uni-modal clusters, then combines KDE estimates for each cluster.
result ROME outperforms state-of-the-art methods and is more robust to various distributions.
This paper proposes MM-DAGs for analyzing traffic congestion, learning multiple DAGs jointly.
problem Analyzing multi-modal traffic data with overlapping and distinct variables.
method Developed MM-DAGs for multi-task, multi-modal DAG learning, using multi-modal regression and CD measure.
result Proved the effectiveness of MM-DAGs in traffic congestion analysis.
ESE-FN improves elderly activity recognition accuracy.
problem Recognizing individual actions and human-object interactions in elderly activities.
method Exploits multi-modal features from RGB videos and skeleton sequences using ESE attentions and a new Multi-modal Loss.
result ESE-FN achieves best accuracy on ETRI-Activity3D dataset.
Study predicts stock price direction on earnings announcement days using multi-modal deep learning.
problem Predicting stock price movements during earnings announcements is challenging due to market noise and discontinuities.
method Constructed a multi-modal feature space combining fundamental metrics, technical indicators, and sentiment scores from financial news articles. Evaluated LSTM and Transformer models against a baseline.
result Transformer model outperforms LSTM in identifying volatile movements, achieving higher macro F1-score.
The paper evaluates samplers on multi-modal targets, focusing on mode separation and recovery.
problem Handling multi-modality in sampling.
method Synthetic experimental setting focusing on mode relative importance recovery.
result Illustrates the challenges and potential of samplers in multi-modality.
Study reveals limits of PLS in multi-modal learning with correlated signals.
problem Understanding PLS performance in multi-modal learning with correlated signals.
method Random matrix theory analysis of spiked cross-covariance models.
result Identifies SNR and correlation regimes where PLS fails to recover any signal.
Sensor fusion has wide applications in many domains including health care and autonomous systems. While the advent of deep learning has enabled promising multi-modal fusion of high-level features and end-to-end sensor fusion solutions, existing deep learning based sensor fusion techniques including deep gating architec…
New method samples from multi-modal distributions on Riemannian manifolds without training.
problem Sampling from multi-modal distributions on Riemannian manifolds is challenging.
method Simulation of a non-equilibrium deterministic dynamics to transport noise toward target distributions.
result Method is entirely training-free and effective on various multi-modal problems.
We propose a novel approach to train a multi-modal policy from mixed demonstrations without their behavior labels. We develop a method to discover the latent factors of variation in the demonstrations. Specifically, our method is based on the variational autoencoder with a categorical latent variable. The encoder infer…
MTRGL learns temporal correlations from multi-modal data for improved pair trading.
problem Discerning temporal correlations among financial entities.
method Combines time series data and discrete features into a temporal graph, using a memory-based temporal graph neural network.
result MTRGL outperforms traditional methods in temporal graph link prediction and pair trading.
New method trains neural samplers to sample from multi-modal distributions efficiently.
problem Mode-seeking behavior of reverse KL divergence hinders effective sampling from multi-modal target distributions.
method Minimizing reverse diffusive KL divergence along diffusion trajectories of model and target densities.
result Demonstrated enhanced sampling performance across various multi-modal distributions.
Increasingly many real world tasks involve data in multiple modalities or views. This has motivated the development of many effective algorithms for learning a common latent space to relate multiple domains. However, most existing cross-view learning algorithms assume access to paired data for training. Their applicabi…
A framework is proposed to detect anomalies in multi-modal data. A deep neural network-based object detector is employed to extract counts of objects and sub-events from the data. A cyclostationary model is proposed to model regular patterns of behavior in the count sequences. The anomaly detection problem is formulate…
IndiSeek learns disentangled representations by balancing independence and completeness.
problem Learning disentangled representations with mutual information in multi-modal data.
method Combines independence-enforcing objective with a reconstruction loss that bounds conditional mutual information.
result Demonstrates effectiveness on synthetic data, CITE-seq, and real-world multi-modal benchmarks.
Piecewise normalizing flows improve multi-modal distribution modeling.
problem Improving accuracy in modeling multi-modal distributions.
method Divide target distribution into clusters, train flows to match standard normal base.
result Piecewise flows outperform standard approaches in accuracy.
Diffusion models enhance robotic manipulation through probabilistic multi-modal learning.
problem Enhancing robotic manipulation through robust and multi-modal learning.
method Probabilistic diffusion models integrating imitation and reinforcement learning.
result Diffusion models improve grasp learning, trajectory planning, and data augmentation in robotics.
Framework for reconstructing nonlinear systems from multi-modal time series data.
problem Reconstructing nonlinear dynamical systems from multi-modal time series data.
method Dynamic interpretable recurrent neural networks coupled with generalized linear models for multi-modal data integration.
result Framework efficiently compensates for noisy or missing information in one data channel using other channels.
MountainLion uses LLMs to interpret financial data and generate investment strategies.
problem Challenges in integrating heterogeneous data for financial trading.
method Multi-modal LLM-based agents that process textual and visual data.
result Improves returns and investor confidence through interpretable investment framework.
The Hamiltonian Monte Carlo (HMC) sampling algorithm exploits Hamiltonian dynamics to construct efficient Markov Chain Monte Carlo (MCMC), which has become increasingly popular in machine learning and statistics. Since HMC uses the gradient information of the target distribution, it can explore the state space much mor…