A new method for feature fusion in U-Net decoders using difference-based gating.
problem Precise fusion of high-level semantics and low-level details in U-Net decoder reconstruction.
method Proposes two difference-based gating approaches: Feature-difference gating (FDG) and Entropy-difference gating (EDG).
result Both FDG and EDG methods outperform existing attention-based fusion methods, with EDG showing superior performance.
A new method selects features for clustering without labels.
problem Identifying meaningful features in large datasets.
method Differentiable unsupervised feature selection using a gated Laplacian.
result The method improves clustering performance in noisy data.
Improved neural networks with conditional gating and reduced computational cost.
problem Training large capacity neural networks with high accuracy and low computational cost.
method Fine-grained channel gating using batch-shaping and residual blocks.
result Conditional gating improves network accuracy and computational efficiency.
ExGate improves neural network accuracy with feature-based attention.
problem Efficiency and accuracy in artificial neural networks for feature-based attention.
method Externally controlled neuron gating in artificial neural networks.
result 5% increase in classification accuracy on CIFAR-10 dataset.
GTI network learns linguistic features for multi-task sequence tagging.
problem Improving neural model performance on multi-task sequence tagging without explicit features.
method GTI network with neural gate modules to learn relations between tasks.
result GTI network outperforms baselines on chunking and NER tasks.
The paper analyzes the role of ReLU gates in deep learning networks.
problem Understanding the role of gates in deep learning networks.
method Developed neural path features (NPF) and neural path values (NPV) to characterize the active sub-networks during training.
result The neural path kernel associated with NPFs is a fundamental quantity that characterizes the information stored in the gates of a DNN.
Improved molecular property prediction using attention and gate mechanisms.
problem Predicting molecular properties from chemical data.
method Attention and gate mechanisms in graph convolutional networks.
result Improved prediction of molecular properties, including photovoltaic efficiency.
Channel gating reduces CNN computation cost by skipping ineffective feature regions.
problem Reducing computation cost in CNNs while maintaining accuracy.
method Dynamic, fine-grained pruning scheme that identifies and skips computation on ineffective feature regions.
result 2.7-8.0x reduction in FLOPs and 2.0-4.4x reduction in memory accesses with minimal accuracy loss.
Optimized deep learning architectures improve sensor fusion performance.
problem Sensor fusion in autonomous systems.
method Proposed two optimized architectures: coarser-grained and two-stage gated.
result Significant performance improvements and robustness in noisy conditions.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
Gated Linear Networks bypass feature learning for fast online learning.
problem Fast online learning and feature learning trade-offs in neural networks.
method Distributed and local credit assignment mechanism, data-dependent gating, online convex optimization.
result GLNs achieve universal learning capabilities and resilience to catastrophic forgetting.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
DIVI clusters noisy high-dimensional data with stable feature gating.
problem Challenging clustering in high-dimensional noisy data.
method Data-informed variational clustering framework combining global feature gating and adaptive structure growth.
result DIVI performs competitively under severe feature noise and remains computationally feasible.
This paper analyzes how gradient descent can learn parameters in mixture-of-experts models with gating.
problem Understanding parameter recovery in mixture-of-experts models with gating.
method Careful analysis of the optimization landscape and design of two distinct loss functions.
result Gradient descent can accurately learn parameters in mixture-of-experts models with gating.
AdaEnsemble learns adaptive feature interactions for CTR prediction.
problem Learning feature interactions for CTR prediction in recommender systems and Ads ranking.
method AdaEnsemble is a Sparsely-Gated Mixture-of-Experts (SparseMoE) architecture that dynamically selects feature interaction depth.
result AdaEnsemble achieves better prediction accuracy and inference efficiency compared to state-of-the-art models.
Study on recurrent neural networks' feature selection and memorization using F1B test.
problem Conflict between feature selection and memorization in sequence learning.
method Flagged-1-Bit (F1B) test, four recurrent network models studied analytically and experimentally.
result Conflict can be resolved by gating mechanism or increasing state dimension.
G-GLN extends GLNs to multiple regression and density modeling.
problem Learning features in deep neural networks.
method G-GLN uses a distributed and local credit assignment mechanism based on optimizing a convex objective.
result G-GLN achieves competitive or state-of-the-art performance on regression benchmarks.
Proposes GPHMEs using Gaussian processes for hierarchical expert models.
problem Hierarchical mixtures of experts with complex gating functions.
method Gaussian process-gated hierarchical mixtures of experts (GPHMEs) with non-linear gating and expert functions.
result Outperforms tree-based HMEs and achieves good performance with reduced complexity.
Study improves stock movement prediction using multimodal data.
problem Inaccurate stock movement prediction due to incomplete multimodal data integration.
method Introduces MSGCA framework for robust multimodal fusion.
result MSGCA framework outperforms existing methods by 21.7% on multimodal datasets.
Proposes a new method for feature selection in non-linear functions.
problem Feature selection for non-linear functions in high-dimensional data.
method Continuous relaxation of Bernoulli distributions to learn feature selection indicators via gradient descent.
result Demonstrates the effectiveness of the approach on synthetic and real-life applications.
Regularized MLE improves MoE models for high-dimensional data.
problem Modeling with high-dimensional predictors and feature selection.
method Gaussian gating network, ℓ1-regularized MLE, EM-Lasso algorithm, BIC-like criterion. result Regularized MLE outperforms standard MLE in clustering and regression tasks.
DeepESNs outperform ESN and GRUs in multivariate time-series prediction.
problem Comparing DeepESNs and gated RNNs for multivariate time-series prediction.
method Experimental comparison of DeepESNs and gated RNNs (Gated Recurrent Units, Long Short-Term Memory) on 4 polyphonic music tasks.
result DeepESNs outperform ESN and GRUs in terms of prediction accuracy and efficiency.
The study examines how character and word-level representations improve sentence-level tasks.
problem Improving the quality of word and sentence representations using character-level information.
method Feature-wise sigmoid gating mechanism for combining character and word-level representations.
result Modeling characters improves final word and sentence representations, especially for less frequent words.
New insights into continual learning with task similarity.
problem Challenges in learning similar tasks without interference.
method Linear teacher-student model with latent structure.
result High input feature similarity with low readout similarity is catastrophic.
GHNet improves graph learning by balancing homogeneity and heterogeneity.
problem Over-smoothing in GCN leads to similar node representations.
method GHNet uses gating units to balance homogeneity and heterogeneity in feature propagation.
result GHNet achieves larger receptive fields without over-smoothing.
MGpi model predicts social actions in group interactions.
problem Social interaction among multiple agents and groups.
method Deep neural network with Kinesic-Proxemic-Message gate for social signal gating.
result Achieves state-of-the-art performance in group identification.
MCFNet recovers spatial detail and fuses it with semantic information for real-time segmentation.
problem Recovering spatial detail information and fusing it with semantic information in real-time.
method Proposes a new architecture (MCFNet) with feature refinement and fusion modules, and a gating unit.
result Achieves competitive performance with high speed (75.5% mIOU, 151.3 FPS on Cityscapes).
Adversarial MoE learns category-specific models for product search.
problem Variations in product features and importance across categories.
method Mixture of Experts with adversarial regularization and soft gating constraints.
result Improved clustering of gate output vectors and shared experts among similar categories.
Improved robustness in multi-modal sensor fusion with deep learning.
problem Inconsistency in fusion weights leading to poor performance under sensor failures.
method Proposes deep multi-modal sensor fusion architectures with fusion weight regularization and target learning.
result Proposed architectures outperform existing deep learning methods under sensor failures.
Theory explains how recurrent networks remember sequences.
problem Understanding how recurrent networks remember sequences and perform well.
method Mean field theory and random matrix theory applied to RNNs with gating mechanisms.
result Gated RNNs outperform non-gated RNNs in remembering sequences.
Improved VQA accuracy with generalized fusion operators.
problem Enhancing multimodal fusion for better VQA performance.
method Generalized Hadamard-Product fusion operators with Nonlinearity Ensembling, Feature Gating, and post-fusion layers.
result 1.1% absolute improvement on VQA 2.0 test-dev set.
KATA improves associative recall by optimizing feature maps derived from nonnegative attention weights.
problem Linear attention's poor performance on associative recall tasks.
method Formulates attention recall as a spherical-packing problem and introduces Kernelized Linear Attention Activations (KATA).
result KATA features offer a favorable capacity-interference tradeoff, enabling efficient associative recall.
Deep neural network learns meaningful factors to predict stock returns.
problem Predicting excess returns of assets like Tesla stock.
method 5-layer deep neural network with gated activation layer to filter noise.
result Proposed model outperforms in predicting stock returns over 2,000 stocks.
New deep learning framework for tabular data clusters with interpretable features.
problem Need for reliable and interpretable clustering models for tabular data.
method Self-supervised feature selection and gate matrix for cluster-level feature selection.
result Model provides interpretable cluster assignments with driving features.
The paper studies 4-qubit Clifford states and their properties.
problem Understanding the set and properties of 4-qubit Clifford states.
method Analyzing the 293760 4-qubit Clifford states, splitting them into 18 groups, and studying the action of CNOT gates and local gates.
result There are 293760 4-qubit Clifford states with specific entanglement entropies, and any pair can be connected with local gates and at most 3 CNOT gates.
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
GD networks improve lung nodule classification accuracy.
problem Difficult classification of lung nodules of varying sizes.
method Proposes Gated-Dilated (GD) networks with Context-Aware sub-network.
result GD network outperforms state-of-the-art models with AUC > 0.95.
New method prepares 3-qubit states using local gates and controlled-Z gates.
problem Preparation of 3-qubit states using quantum gates.
method Uses Ry(θ) gates and controlled-Z gates, with an optimal number of controlled-Z gates. result Optimal number of controlled-Z gates for preparing 3-qubit states is four. GIB improves neural network generalization by dynamically selecting task-relevant features across different sequential environments.
problem Poor generalization of deep neural networks to unseen environments.
method Proposes a gated information bottleneck (GIB) approach that dynamically drops spurious correlations and selects task-relevant features.
result GIB outperforms other IB approaches in adversarial robustness and OOD detection.
A forget-gate-only LSTM outperforms standard LSTM on benchmark datasets.
problem The necessity of all gates in LSTM networks.
method A forget-gate-only LSTM with chrono-initialized biases.
result The forget-gate-only LSTM outperforms standard LSTM on MNIST and pMNIST datasets.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
In quantum computation, series of quantum gates have to be arranged in a predefined sequence that led to a quantum circuit in order to solve a particular problem. What if the sequence of quantum gates is known but both the problem to be solved and the outcome of the so defined quantum circuit remain in the shadow? This…
Proposes a new LSTM gate structure using bivariate Beta distribution.
problem Inflexibility of sigmoid gates in modeling multi-modality and skewness, and lack of modeling correlation between gates.
method Introduces a bivariate Beta distribution gate structure within LSTM cells.
result Empirically shows higher gradient values and improved model performance.
Bayesian method sparsifies gated RNNs, improving speed and interpretability.
problem Sparsifying neural networks to reduce complexity and improve performance.
method Bayesian approach to sparsify weights, neurons, and gates in LSTM architectures.
result Sparsified gated RNNs speed up forward pass and improve compression.
CycleGAN-VC converts speech without parallel data, achieving high quality.
problem Voice conversion without parallel data.
method Cycle-Consistent Adversarial Networks (CycleGAN) with gated CNNs and identity-mapping loss.
result Converted speech quality comparable to Gaussian mixture model-based methods under advantageous conditions.
We sparsify gated RNNs by simplifying their structure.
problem Improving efficiency of RNNs by reducing their complexity.
method Adjust existing sparsification techniques to gated RNNs, sparsifying preactivations of gates.
result Simplified LSTM structure improves model performance and efficiency.
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
Improved RNNs with flexible gates using kernel activation functions.
problem Modeling long-term dependencies in sequential data.
method Designed a more flexible architecture with adaptable parameters using kernel activation functions.
result Improved accuracy with negligible computational cost and speed-up in training iterations.