Improved robustness in multi-modal sensor fusion with deep learning.
problem Inconsistency in fusion weights leading to poor performance under sensor failures.
method Proposes deep multi-modal sensor fusion architectures with fusion weight regularization and target learning.
result Proposed architectures outperform existing deep learning methods under sensor failures.
MEx dataset benchmarks HAR and multi-modal fusion for exercise quality.
problem Recognizing and evaluating exercise quality for Musculoskeletal Disorders patients.
method Multi-sensor, multi-modal dataset with four sensors (pressure mat, depth camera, accelerometers) for HAR and exercise quality assessment.
result Reference performance for each sensor identified, exposing their strengths and weaknesses.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
Adversarial approach enhances sensor fusion for robust target detection.
problem Improving target detection and classification using multi-modal sensor fusion.
method Generative network learns latent space from various sensor modalities, then detects damaged sensors and safeguards performance.
result Automatic robustness against noisy/damaged sensors achieved.
A deep network learns diverse contexts from multi-modal sensor data.
problem Recognizing diverse contexts and activities from multi-modal sensor data.
method Multi-stream temporal convolutional network with contextualization module.
result Deep network achieves optimal recognition rate.
Improved sleep apnea detection using sensor fusion and backward shortcut connections.
problem Untreated sleep apnea leads to severe health consequences; automated detection is needed.
method Late sensor fusion using backward shortcut connections to improve deep learning models.
result Significant improvement in predictive performance over single sensor methods.
System detects social interactions in crowds using mobile phone sensors.
problem Detecting social interactions in crowded settings.
method Multi-modal mobile sensing (BLE, accelerometer, gyroscope) and machine learning.
result 77.8% precision and 86.5% recall for predicting social interactions.
Cooperative perception improves 3D object detection in autonomous vehicles.
problem Limited field-of-view and occlusion in single sensor data.
method Early fusion of point clouds from multiple sensors, late fusion of independently detected bounding boxes, and hybrid combination.
result Early fusion approach outperforms late fusion by significantly higher recall (95%) compared to single-point sensing (30%).
New approach improves human activity recognition with wearables.
problem Improving human activity recognition with wearables.
method Exploiting latent relationships between multi-channel sensor modalities, data-agnostic augmentation, and a classification loss criterion.
result Achieves new state-of-the-art performance on four diverse activity recognition benchmarks.
This paper develops a mathematical and computational framework for analyzing the expected performance of Bayesian data fusion, or joint statistical inference, within a sensor network. We use variational techniques to obtain the posterior expectation as the optimal fusion rule under a deterministic constraint and a quad…
TACTO simulates high-resolution touch sensing for robotics.
problem Accurate simulation of touch sensing in robotics.
method Fast, flexible, open-source simulator for vision-based tactile sensors.
result Demonstrated TACTO's effectiveness in grasping stability prediction and marble manipulation control.
Model traffic congestion events using multi-modal data and attention-based neural networks.
problem Capture non-homogeneous temporal and directional spatial dependencies in traffic congestion events.
method Attention-based neural networks for point processes, adapted tail-up model for spatial statistics.
result Superior performance compared to state-of-the-art methods on synthetic and real data.
New method fuses audio and magnetic data to identify underlying subspaces.
problem Identifying complex trends in multi-modality data.
method Robust Group Subspace Recovery (RoGSuRe) algorithm based on group sparsity and bi-sparsity pursuit.
result Competitive performance in clustering and classification of multi-modal data.
Enhances mobile context prediction using weakly supervised learning.
problem Power consumption and limited training data for always-on context prediction.
method Weakly supervised learning framework for multi-modal sensing.
result Personalized context prediction model with improved accuracy.
The paper analyzes and proposes an algorithm for multi-modal nonlinear embeddings with theoretical performance bounds.
problem Generalizability of multi-modal nonlinear embeddings to unseen data.
method Theoretical analysis and a multi-modal nonlinear representation learning algorithm motivated by performance bounds.
result The proposed algorithm yields promising performance in multi-modal image classification and cross-modal image-text retrieval applications.
ROME improves density estimation for multi-modal, non-normal data.
problem Robust multi-modal density estimation in non-normal, highly correlated distributions.
method ROME uses clustering to segment multi-modal data into uni-modal clusters, then combines KDE estimates for each cluster.
result ROME outperforms state-of-the-art methods and is more robust to various distributions.
The paper evaluates samplers on multi-modal targets, focusing on mode separation and recovery.
problem Handling multi-modality in sampling.
method Synthetic experimental setting focusing on mode relative importance recovery.
result Illustrates the challenges and potential of samplers in multi-modality.
AF improves sampling from high-dimensional, multi-modal distributions.
problem Sampling from high-dimensional, multi-modal distributions is challenging.
method Annealing Flow (AF) using Continuous Normalizing Flow (CNF) with dynamic Optimal Transport (OT) objective and annealing procedures.
result AF significantly improves training efficiency and stability, outperforming state-of-the-art methods.
Develops multi-modal neural network models for improved prediction and uncertainty quantification.
problem Improving prediction accuracy and uncertainty quantification for multi-modal data.
method Multi-modal Bayesian neural network models with conjugate last-layer estimation using SVI.
result Improved prediction accuracy and uncertainty quantification compared to uni-modal models.
METEOR learns efficient representations from multi-modal data streams.
problem Efficiently interpreting multi-modal information in complex environments.
method METEOR learns compact representations by sharing parameters within semantically meaningful groups and preserving domain-agnostic semantics.
result METEOR reduces memory usage by around 80% compared to conventional methods.
Framework for handling long-tailed multi-modal data.
problem Class imbalance and long-tailed distributions in multi-modal data.
method Multi-expert architecture with modality-specific networks and dynamic fusion weights.
result Framework outperforms existing methods in long-tailed, class-imbalanced scenarios.
New framework shows cross-attention improves multi-modal in-context learning.
problem Understanding multi-modal in-context learning in neural networks.
method Mathematical framework and linearized cross-attention mechanism.
result Cross-attention mechanism is provably optimal for multi-modal in-context learning.
A novel multi-modal active learning approach using RL for engagement estimation.
problem Challenges in labeling multi-modal human data for accurate user state estimation.
method Deep reinforcement learning for optimal data selection and multi-modal data fusion.
result The proposed approach outperforms existing methods in engagement estimation.
Inductive graph-based approach for disease classification with incomplete data.
problem Classifying patients with incomplete multi-modal data.
method Multi-modal graph fusion trained end-to-end for node-level classification.
result Outperforms single static graph approach in multi-modal disease classification.
This paper improves sentiment classification by combining text, audio, and video data using DCCA.
problem Improving sentiment classification accuracy using multi-modal data.
method Deep Canonical Correlation Analysis (DCCA) for combining text, audio, and video embeddings.
result One-Step DCCA outperforms current state-of-the-art in multi-modal embedding learning.
Contrastive learning adapts to data intrinsic dimensions, learning low-dimensional representations.
problem Learning high-dimensional representations from multi-modal data.
method Multi-modal contrastive learning with temperature optimization.
result Contrastive learning adapts to intrinsic dimensions of data, not specified dimensions.
Adaptive anchor methods improve multi-modal learning by balancing intra-modal and inter-modal information.
problem Fixed anchor methods limit multi-modal learning by over-reliance on a single modality and inadequate cross-modal correlation.
method Adaptive anchor methods using centroid-based anchors from all modalities.
result Adaptive anchor methods like CentroBind consistently outperform fixed anchor methods across various datasets.
PyTorch Frame simplifies multi-modal tabular learning with modular data and model handling.
problem Handling complex multi-modal tabular data in deep learning.
method A PyTorch-based framework that provides a data structure, model abstraction, and integration with external models.
result Demonstrated the effectiveness of PyTorch Frame in implementing and applying diverse tabular models to complex multi-modal tabular data.
Paper proposes multi-task learning for multi-modal video Q&A.
problem Expensive to create large-scale datasets for multi-modal video Q&A.
method Composed of three networks: video Q&A, temporal retrieval, and modality alignment.
result State-of-the-art results on TVQA dataset.
Derives M2VAE objective from marginal joint log-likelihood.
problem Training Multi-Modal Variational Autoencoders (M2VAEs). method Derives trainable evidence lower bound from marginal joint log-likelihood.
result Derives M2VAE objective from marginal joint log-likelihood. Method trains multi-modal policy from unlabeled mixed demonstrations.
problem Training policies from unlabeled mixed demonstrations.
method Variational autoencoder with categorical latent variable to discover latent factors of variation.
result Policy can reproduce specific behaviors by conditioning on categorical vectors.
This work improves multi-modal generative models by using permutation-invariant neural networks.
problem Improving multi-modal generative models with tighter variational objectives.
method Developed more flexible aggregation schemes based on permutation-invariant neural networks.
result Our variational objective and flexible aggregation models can better approximate the true joint distribution.
MMVAE learns multi-modal data with shared and private latent spaces.
problem Learning useful representations across multiple data modalities.
method Mixture-of-experts variational autoencoder (MMVAE).
result MMVAE satisfies four criteria for multi-modal learning.
This paper reviews deep learning for multi-modality medical image segmentation.
problem Improving segmentation accuracy in medical images using multiple modalities.
method Overview of deep learning and multi-modal medical image segmentation, analysis of different network architectures and fusion strategies.
result Later fusion of modalities can lead to more accurate segmentation results.
Method predicts NBA players' multi-modal movement trajectories.
problem Understanding NBA players' decision-making during games.
method LSTM-based architecture with multi-modal loss function.
result Method outperforms state-of-the-art in predicting realistic trajectories.
This work builds a sensor graph from DC sensors for anomaly detection.
problem Anomaly detection in data centers with complex sensor relationships.
method Data-driven pipeline (ts2graph) to build a sensor graph from sensor time series.
result Graph neural network (GNN) outperforms existing methods by 2-3 times in anomaly detection.
A new MCMC method improves HMC for distant multi-modal sampling.
problem Sampling from multi-modal distributions, especially when modes are far apart.
method Improves Hamiltonian dynamics and uses variational distribution.
result The method can converge to target distributions effectively.
Graph convolution model uses self-attention to predict diseases from multi-modal data.
problem Predicting diseases from diverse multi-modal data.
method Graph convolution with self-attention layer.
result Significantly outperforms state-of-the-art methods in disease prediction.
This study optimizes multi-modal learning thresholds and algorithms in high dimensions.
problem Optimizing multi-modal learning performance in high-dimensional data.
method Analytical quantification and derivation of AMP algorithm with state evolution analysis.
result Bayes-optimal performance and recovery thresholds derived for multi-modal data.
ESE-FN improves elderly activity recognition accuracy.
problem Recognizing individual actions and human-object interactions in elderly activities.
method Exploits multi-modal features from RGB videos and skeleton sequences using ESE attentions and a new Multi-modal Loss.
result ESE-FN achieves best accuracy on ETRI-Activity3D dataset.
MCC algorithm predicts with partial modalities, outperforming full modalities.
problem Predicting with inconsistent and diverse multi-modal data.
method Instance-oriented Multi-modal Classifier Chains (MCC) algorithm.
result MCC outperforms full modalities in prediction.
Paper introduces topological eigenvalue theorems for tensor analysis in multi-modal data.
problem Lack of deep understanding of tensor structures in multi-modal data fusion.
method Introduces topological perspective to tensor eigenvalue analysis, linking eigenvalues to topological features.
result Establishes new theorems that enhance understanding of tensor structures in data fusion.
GPCCA integrates multi-modal data with missing values, improving clustering accuracy.
problem Integrating and analyzing multi-modal data with missing values and partial observations.
method Generalized Probabilistic Canonical Correlation Analysis (GPCCA) for unsupervised multi-modal data integration and dimensionality reduction.
result GPCCA outperforms existing methods in capturing essential patterns across modalities and provides robust low-dimensional embeddings.
Proposes a novel method for detecting novelty in multi-modal data.
problem Challenges in detecting novelty in high-dimensional, multi-modal data.
method Orthogonalized latent space for disentangling features and defining novelty score.
result Proposed method outperforms state-of-the-art algorithms in novelty detection.
CF-VAE models capture multi-modal distributions for better structured sequence prediction.
problem Challenges in capturing multi-modality of future states in latent variable models.
method Conditional Flow Variational Autoencoders (CF-VAE) with conditional normalizing flows.
result CF-VAE achieves state-of-the-art results on multi-modal structured sequence prediction datasets.
New divergences improve score-based methods for multi-modal distributions.
problem Blindness problem in score-based divergences for multi-modal distributions.
method Proposed a new family of divergences to mitigate blindness.
result Improved performance in density estimation compared to traditional approaches.
DeepLocalization uses neural networks to localize vehicles from landmarks.
problem Vehicle self-localization from multi-modal sensor data and a reference map.
method Deep neural network that regresses vehicle pose from unordered landmarks.
result DeepLocalization achieves state-of-the-art accuracy and is faster than related work.
Study uses sentiment analysis to predict cryptocurrency token returns in virtual reality.
problem Predicting cryptocurrency token returns in virtual reality economies.
method Used BERT for sentiment analysis and developed LSTM models integrating multi-modal features.
result Multi-modal model significantly outperforms price-only baseline in prediction accuracy.