Partial fusion combines neural networks to balance accuracy and efficiency.
problem Balancing accuracy and computational cost in neural networks.
method Extending weight aggregation methods based on neuron-level similarity, using partial optimal transport to match similar neurons.
result Achieves a flexible tradeoff between computational cost and performance.
New model learns execution of code using GNNs.
problem Stagnation of computer system performance due to Moore's Law.
method Multi-task GNN over low-level code and program state.
result Improved performance on dynamic tasks (26% and 45% over state-of-the-art).
Neural surrogates speed up 5D gyrokinetic simulations of plasma turbulence.
problem Expensive numerical simulations of plasma turbulence hinder fusion reactor design.
method Trained a hierarchical vision transformer in 5D to predict plasma quantities faster.
result Neural surrogates predict plasma quantities two orders of magnitude faster than numerical codes.
New method learns fusion rules from few images using granular ball priors.
problem Challenges in supervised learning for image fusion with limited data.
method Introduces incomplete priors and Granular Ball Pixel Computation (GBPC) algorithm.
result Lightweight neural network learns effective fusion rules from few images.
Improved neural network surrogates for ICF using manifold and cycle consistency.
problem Modeling and predicting complex physical processes in inertial confinement fusion.
method Training neural network surrogates that are consistent with the physical manifold and cyclically consistent.
result Surrogates are superior in predictive performance, more resilient to sampling artifacts, and more data efficient.
Optimal transport aligns model neurons for efficient fusion.
problem Resource constraints in ensemble model averaging.
method Layer-wise model fusion using optimal transport.
result Softly aligns neurons across models for efficient knowledge transfer.
Fusion of transformer networks using optimal transport for improved performance.
problem Improving performance of transformer-based models through fusion.
method Exploiting optimal transport for soft alignment of transformer components.
result Consistently outperforms vanilla fusion and individual parent models.
Proposes Fusion Recurrent Neural Network for sequence data.
problem Improving sequence learning for practical applications.
method Fusion module and Transport module for sequence data.
result Fusion RNN performs comparably to state-of-the-art RNNs.
Paper proposes a novel approach to improve spatiotemporal precipitation forecasts.
problem Improving accuracy of spatiotemporal precipitation forecasts for flood damage mitigation.
method Introduces a rain-code fusion approach using ConvLSTM and multi-frame fusion for spatiotemporal precipitation code-to-code forecasting.
result Demonstrates enhanced accuracy in precipitation forecasts beyond 3 timesteps using the rain-code fusion.
This work uses scientific constraints to validate neural network predictions in fusion physics.
problem Verifying the scientific plausibility of neural network predictions in fusion physics.
method Using known scientific constraints as a validation tool.
result Validated neural network predictions in fusion physics using scientific constraints.
Meta Fusion integrates various multimodal data fusion strategies into a unified framework.
problem Improving predictive power of machine learning methods across diverse applications.
method Meta Fusion constructs a cohort of models based on latent representations across modalities, sharing soft information to boost performance.
result Meta Fusion consistently outperforms conventional fusion strategies in simulation and real-world applications.
Two solutions for multi-modal record linkage using Deep Learning inspired by Visual Question Answering.
problem Matching records from multiple sources representing the same entity.
method Two fusion modules: Recurrent Neural Network + Convolutional Neural Network and Stacked Attention Network. A Siamese Neural Network computes similarity.
result Recurrent Neural Network + Convolutional Neural Network fusion module outperforms a simple model.
Deep CNN predicts disruptions in fusion plasmas with high accuracy.
problem Predicting plasma events in fusion devices with multi-scale, multi-physics characteristics.
method Deep convolutional neural networks (CNN) with dilated convolutions trained on ECEi diagnostic data.
result Deep CNN achieves an F1-score of ~91% on disruption prediction.
Layer fusion reduces deep neural network layers with minimal loss in accuracy.
problem Model compression to reduce neural network size and computation.
method Fusion of similar layers to reduce model size with minimal performance loss.
result Deep networks can be compressed up to 3.33x with minimal accuracy loss.
We propose a generalized class of multimodal fusion operators for the task of visual question answering (VQA). We identify generalizations of existing multimodal fusion operators based on the Hadamard product, and show that specific non-trivial instantiations of this generalized fusion operator exhibit superior perform…
In this paper, we propose to employ a bank of modality-dedicated Convolutional Neural Networks (CNNs), fuse, train, and optimize them together for person classification tasks. A modality-dedicated CNN is used for each modality to extract modality-specific features. We demonstrate that, rather than spatial fusion at the…
New fusion blocks improve equivariant neural networks for molecular dynamics.
problem Designing equivariant neural networks for tasks with global symmetries.
method Using fusion diagrams from tensor networks to design novel equivariant components.
result Improved performance with fewer parameters on chemical problems.
LDF combines neural networks with probabilistic models for data fusion.
problem Combining limited primary data with readily available auxiliary data.
method Neural networks as conjugate mappings of auxiliary data for posterior analysis.
result Efficient inference and compact latent variable posterior distributions.
Sensor fusion is a key technology that integrates various sensory inputs to allow for robust decision making in many applications such as autonomous driving and robot control. Deep neural networks have been adopted for sensor fusion in a body of recent studies. Among these, the so-called netgated architecture was propo…
PG-EVIKAL refines molecular property predictions using neighbor fusion and evidential neural networks.
problem Improving molecular property predictions using test-time neighbor fusion.
method Adapting evidential neural networks to refine predictions by re-ranking structurally similar neighbors.
result PG-EVIKAL reduces RMSE on 14 out of 16 molecular datasets, improving calibration and sequential refinement.
Paper proposes a new framework for robust multi-modal data fusion under uncertainty.
problem Unexpected modality failures in nonlinear non-Gaussian dynamic processes.
method Dynamic model averaging (DMA) based particle filter (PF) algorithm.
result The proposed solution outperforms state-of-the-art methods in experiments.
New deep fusion methods improve human action recognition using depth and inertial sensor data.
problem Existing multimodal HAR frameworks lack mid-level feature fusion.
method Proposes three deep multilevel multimodal fusion frameworks, transforming depth and inertial sensor data into images and using convolution with Prewitt filter to create modality within modality.
result Supremacy of proposed fusion frameworks over existing methods on three publicly available datasets.
A new method merges neural networks using CCA to improve model performance.
problem Improving model accuracy through ensembling while reducing computational and storage costs.
method CCA Merge, a new model merging algorithm based on Canonical Correlation Analysis.
result CCA Merge leads to better model performance than past methods, especially in merging more than two models.
A new multi-view clustering method that is fast, scalable, and easy to use.
problem High computational complexity, one-stage fusion, and dataset-specific hyperparameter tuning in multi-view clustering.
method Random view groups, hybrid early-late fusion, diversified base clusterings, and unified bipartite graph.
result Almost linear time and space complexity, no dataset-specific tuning required.
GWN improves multimodal data fusion accuracy for chronic pain patients.
problem Dynamic and unspecified uncertainties in multimodal data fusion.
method Inspired by Global Workspace Theory, GWN is a neural network architecture that dynamically attends to multiple modalities.
result GWN achieved higher F1 scores (0.92 and 0.75) for multimodal discrimination and classification tasks.
Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the in…
In this paper, we propose a deep multimodal fusion network to fuse multiple modalities (face, iris, and fingerprint) for person identification. The proposed deep multimodal fusion algorithm consists of multiple streams of modality-specific Convolutional Neural Networks (CNNs), which are jointly optimized at multiple fe…
Automatic speech recognition can potentially benefit from the lip motion patterns, complementing acoustic speech to improve the overall recognition performance, particularly in noise. In this paper we propose an audio-visual fusion strategy that goes beyond simple feature concatenation and learns to automatically align…
Method uses neural networks to solve combinatorial problems.
problem Solving combinatorial problems on raw input data.
method Integrates blackbox combinatorial solvers into neural networks.
result Efficient backward pass through blackbox solvers implemented.
MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.
problem Designing architectures for diverse tasks with varying priors.
method Disentangled GP-MTL networks, hierarchical feature sharing, and gradient-based search.
result General-purpose model trained once can adapt to multiple tasks.
Neural network fusion reduces data acquisition costs for multi-fidelity sources.
problem Reducing cost in acquiring information from multiple data sources with varying fidelity.
method Employing a novel neural network architecture for nonlinear manifold learning of multi-fidelity data.
result Our approach provides high predictive power and quantifies various sources uncertainties.
A fusion approach combines audio and video features for emotion recognition.
problem Continuous emotion recognition using both visual and auditory modalities.
method Pre-trained CNN features from video frames and minimalistic auditory descriptors. Fusion at feature or prediction level. SVR for prediction.
result Improves CCCs of 0.749 and 0.565 for arousal and valence respectively.
This paper improves model fusion by training-time neuron alignment, reducing barriers in multi-model fusion.
problem Diverse neuron permutations across different settings hinder model fusion performances.
method Training-time neuron alignment using fixed neuron anchors to reduce training-time permutations.
result Training-time neuron alignment improves fusion of pretrained models and federated learning performances.
FuseInit optimizes neural network initialization by fusing layers from deeper networks.
problem Initialization sensitivity of neural networks.
method Fusing neighboring layers from deeper networks trained with random initialization.
result Shallower networks initialized with FuseInit can perform as well as deeper networks.
Neural networks outperform conventional filters in inertial sensor-based attitude estimation.
problem Limited accuracy in inertial sensor-based attitude estimation due to dynamic and static motion.
method Investigated neural networks versus conventional filters for improving accuracy.
result Neural networks outperform conventional filters only with domain-specific optimizations.
Proposes a model combining graph networks and variational Bayes for graph data.
problem Probabilistic modeling of graph structured data.
method Combines graph networks and variational Bayes for probabilistic modeling of graph data.
result Demonstrates effectiveness on wind farm monitoring and Gaussian Process data.
Sparse matrix decomposition identifies key design variables for ICF experiments.
problem Improving predictive capability of ICF simulation codes through better understanding of design inputs and outcomes.
method Sparse Principal Component Analysis (SPCA) and Random Forest (RF) surrogate model.
result Identified clusters of design variables related to physical processes, revealing important variables not previously considered.
Inspired by brain's modality fusion, this paper detects active speakers from audio and video.
problem Detecting active speakers in noisy environments.
method Inspired by brain's superior colliculus, combines audio and visual data through specialized neural networks and a novel fusion layer.
result Achieved results greatly surpassing initial expectations, confirming the effectiveness of the proposed method.
HSACC improves multi-view clustering of incomplete data.
problem Challenges in clustering incomplete multi-view data.
method Hierarchical Semantic Alignment and Cooperative Completion framework.
result HSACC outperforms state-of-the-art methods on benchmark datasets.
Improved speech recognition model with better performance.
problem Speech recognition accuracy on Librispeech.
method Integrates an external language model with an internal LM correction.
result Over 14% relative improvement in performance.
We present convolutional neural network (CNN) based approaches for unsupervised multimodal subspace clustering. The proposed framework consists of three main stages - multimodal encoder, self-expressive layer, and multimodal decoder. The encoder takes multimodal data as input and fuses them to a latent space representa…
One of the core tasks in multi-view learning is to capture relations among views. For sequential data, the relations not only span across views, but also extend throughout the view length to form long-term intra-view and inter-view interactions. In this paper, we present a new memory augmented neural network model that…
AI model predicts stock prices using social media data and hybrid neural networks.
problem Predicting stock price movements during the COVID-19 pandemic.
method Integrates social media trends and historical stock data using a hybrid CNN-BLSTM framework.
result The proposed framework outperforms traditional models in predicting stock price movements.
Human Activity Recognition (HAR) based on motion sensors has drawn a lot of attention over the last few years, since perceiving the human status enables context-aware applications to adapt their services on users' needs. However, motion sensor fusion and feature extraction have not reached their full potentials, remain…
Improved accuracy in dynamic response variation analysis using multi-fidelity data fusion.
problem Inefficient characterization of dynamic response variation due to limited high-fidelity data.
method Composite Neural Network fusion approach for multi-level, heterogeneous datasets.
result Improved accuracy in frequency response variation characterization.
Growing amounts of online user data motivate the need for automated processing techniques. In case of user ratings, one interesting option is to use neural networks for learning to predict ratings given an item and a user. While training for prediction, such an approach at the same time learns to map each user to a vec…
In this paper, we propose a method using a three dimensional convolutional neural network (3-D-CNN) to fuse together multispectral (MS) and hyperspectral (HS) images to obtain a high resolution hyperspectral image. Dimensionality reduction of the hyperspectral image is performed prior to fusion in order to significantl…
Investigates neural codes and their embeddings, proving conjectures and introducing new code types.
problem Analyzing neural codes and their embedding dimensions.
method Combinatorial, topological, and algebraic analysis; proving conjectures; introducing new neural code types.
result Proves conjectures about neural codes and their embeddings, introduces new code types.