The study tightens risk bounds for mixtures of experts using local differential privacy.
problem Improving risk bounds for mixtures of experts.
method Imposing local differential privacy (LDP) on the gating mechanism of mixtures of experts.
result Theoretical bounds exhibit logarithmic dependence on the number of experts and tighter than existing bounds.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
Gradient Gating improves deep GNNs by modulating message passing updates.
problem Oversmoothing and performance degradation in deep GNNs.
method Gradient gating mechanism for multi-rate message passing.
result G2 framework alleviates oversmoothing and achieves state-of-the-art performance. The study examines how character and word-level representations improve sentence-level tasks.
problem Improving the quality of word and sentence representations using character-level information.
method Feature-wise sigmoid gating mechanism for combining character and word-level representations.
result Modeling characters improves final word and sentence representations, especially for less frequent words.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
ExGate improves neural network accuracy with feature-based attention.
problem Efficiency and accuracy in artificial neural networks for feature-based attention.
method Externally controlled neuron gating in artificial neural networks.
result 5% increase in classification accuracy on CIFAR-10 dataset.
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Bayesian models combine experts with a flexible gating mechanism for complex data.
problem Theoretical properties of Bayesian mixture-of-experts models with softmax gating remain unexplored.
method Investigated asymptotic behavior of posterior distribution for density estimation, parameter estimation, and model selection.
result Established posterior contraction rates for density estimation and parameter estimation, providing insights for practical model design.
A new method selects features for clustering without labels.
problem Identifying meaningful features in large datasets.
method Differentiable unsupervised feature selection using a gated Laplacian.
result The method improves clustering performance in noisy data.
Theory explains how recurrent networks remember sequences.
problem Understanding how recurrent networks remember sequences and perform well.
method Mean field theory and random matrix theory applied to RNNs with gating mechanisms.
result Gated RNNs outperform non-gated RNNs in remembering sequences.
Improved GRU model with weighted time-delay feedback for long-term dependencies.
problem Modeling long-term dependencies in sequential data.
method Introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism.
result τ-GRU outperforms state-of-the-art models on various tasks.
LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections. We present an alternative view to explain the success of LSTMs: the gates themselves are versatile recurrent models that provide more representational power than previously appreciated. We do…
This study analyzes communication constraints in MoE architectures using information theory.
problem Communication constraints in Mixture-of-Experts (MoE) architectures.
method Developed a rate-distortion characterization of finite-rate gating in MoE architectures using information theory.
result Yielded capacity-aware limits for communication-constrained MoE systems.
Gated Linear Networks bypass feature learning for fast online learning.
problem Fast online learning and feature learning trade-offs in neural networks.
method Distributed and local credit assignment mechanism, data-dependent gating, online convex optimization.
result GLNs achieve universal learning capabilities and resilience to catastrophic forgetting.
Paper shows how LSTM can remember long sequences by attending to persisted information.
problem LSTMs struggle with long sequences due to fading information and bias towards recent data.
method The paper introduces a mechanism that allows LSTMs to attend to information in memory based on how long it was persisted by the gating mechanism.
result The method improves LSTM's ability to process long sequences by retrieving information proportionally to its persistence in memory.
Simple construction for universal quantum gates.
problem Designing efficient quantum gates for topological computers.
method Demonstrated a simple construction for unitary solutions of the braided Yang-Baxter equation in any dimension.
result Proved the existence of universal quantum gates in any dimension.
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
Paper interprets ResNets via gate-network controls and deep-layer classifications.
problem Understanding the performance mechanism of ResNets.
method Constructs typical solutions using gate-network controls and deep-layer classifications.
result Proves the universal-approximation capability of ResNets.
FC-GAGA forecasts traffic using a novel gating mechanism.
problem Forecasting multivariate time-series, especially with graph relationships.
method Learnable fully connected hard graph gating mechanism for fully connected time-series forecasting.
result Competitive or better performance than existing algorithms without graph knowledge.
Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. We prove that learnable gates in a…
SG-NTF completes HDI tensors with spectral mapping and spatio-temporal gating.
problem High-dimensional and incomplete tensor completion.
method Spectra-Guided Neural Tucker Factorization (SG-NTF) with Spatio-Temporal Co-Gating (STCG).
result Maintains competitive completion accuracy with parameter efficiency.
Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mecha…
New interpretation of RNN forget gate improves learnability for long-term sequential data.
problem Improving learnability of recurrent neural networks for long-term temporal dependencies.
method Generalized theory of gated RNNs, focusing on gradient behavior over time.
result Existing RNNs satisfy the gradient condition for initial training, suggesting validity of forget gate interpretation.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
Molecular structure-property relationships are key to molecular engineering for materials and drug discovery. The rise of deep learning offers a new viable solution to elucidate the structure-property relationships directly from chemical data. Here we show that the performance of graph convolutional networks (GCNs) for…
Recurrent Neural Networks (RNNs) play a major role in the field of sequential learning, and have outperformed traditional algorithms on many benchmarks. Training deep RNNs still remains a challenge, and most of the state-of-the-art models are structured with a transition depth of 2-4 layers. Recurrent Highway Networks …
Novel complex RNN improves stability and performance in sequence tasks.
problem Lack of complex representations in deep learning for sequence tasks.
method Developed a complex gated recurrent cell combining complex-valued and norm-preserving state transitions with a gating mechanism.
result Improves stability and convergence properties, performs competitively on various tasks.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
The paper analyzes convergence rates of softmax gating in MoE models.
problem The effectiveness and scalability of machine learning models using MoE.
method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.
result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.
Flexible priors improve VAE-based CF models for better user preference modeling.
problem Simplistic priors in VAEs limit user preference modeling and deeper representation learning.
method Incorporated flexible priors and gating mechanisms into VAEs for collaborative filtering.
result Flexible priors and gating mechanisms significantly improve recommendation performance.
VGE provides a practical approach to uncertainty estimation in ensemble models.
problem Uncertainty estimation in ensemble models using additive decomposition breaks down.
method Variance-Gated Ensembles (VGE) introduces a differentiable framework with a signal-to-noise gate.
result VGE provides a Variance-Gated Margin Uncertainty (VGMU) score and Variance-Gated Normalization (VGN) layer.
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
MGpi model predicts social actions in group interactions.
problem Social interaction among multiple agents and groups.
method Deep neural network with Kinesic-Proxemic-Message gate for social signal gating.
result Achieves state-of-the-art performance in group identification.
Partially observable environments present an important open challenge in the domain of sequential control learning with delayed rewards. Despite numerous attempts during the two last decades, the majority of reinforcement learning algorithms and associated approximate models, applied to this context, still assume Marko…
Temporal Functional Circuits explain KAN forecasts with interpretable edge functions.
problem Lack of mechanistic explanations in KAN forecasting.
method Transform KAN edge functions into faithful, temporally grounded explanations using a gated residual KAN.
result Gated KAN achieves lower MSE than linear-only models on regime-switching signals.
Study combines CNNs and LSTMs for ECG classification, improving performance with attention mechanisms.
problem Improving ECG classification accuracy for weakly supervised tasks.
method Investigates ConvLSTM networks and various attention mechanisms to capture both morphological and temporal features.
result Attention mechanisms improve classification performance and provide interpretable saliency maps.
LSTM improves cross-network recommendations by capturing user preference changes and irregular time intervals.
problem Offline cross-network recommender solutions fail to capture user preference changes and dynamic environments.
method Proposes a multi-layered LSTM network with attention mechanisms, higher order interactions, and time-aware gates.
result The model consistently outperforms state-of-the-art in accuracy, diversity, and novelty.
G-GLN extends GLNs to multiple regression and density modeling.
problem Learning features in deep neural networks.
method G-GLN uses a distributed and local credit assignment mechanism based on optimizing a convex objective.
result G-GLN achieves competitive or state-of-the-art performance on regression benchmarks.
Recurrent Neural Networks (RNNs) with sophisticated units that implement a gating mechanism have emerged as powerful technique for modeling sequential signals such as speech or electroencephalography (EEG). The latter is the focus on this paper. A significant big data resource, known as the TUH EEG Corpus (TUEEG), has …
We present a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory. We achieve this by extending unitary RNNs with a gating mechanism. Our model is able to outperform LSTM…
WAVE improves time series forecasting by integrating AR and MA components.
problem Time series forecasting challenges.
method WAVE attention mechanism with AR and MA components.
result WAVE attention consistently improves TSF performance.
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
A framework for modular training of robust generative models.
problem Training large generative models is resource-intensive and requires heuristic tuning.
method Modular training using a gating mechanism and a minimax game to find a robust gate.
result The modular approach can theoretically outperform monolithic baselines and is scalable.
Focused RNNs improve sequence processing tasks with attention mechanisms.
problem Sequence processing tasks benefit from more focused attention mechanisms.
method Focused hierarchical RNNs with discrete gating mechanisms that selectively attend to relevant parts of the input.
result Focused RNNs outperform existing methods on various sequence processing tasks, including MS MARCO and SearchQA.
Entropy regularization improves sparse model discovery in federated learning.
problem Sparse model discovery in federated learning with limited data.
method Entropy regularization of gate distributions for probabilistic sparse model exploration.
result Entropy regularization leads to better sparse model recovery and performance.
Dropout helps a stable network learn new tasks without forgetting old ones.
problem Catastrophic forgetting in neural networks when learning multiple tasks.
method Investigate the relationship between dropout and stability in neural networks, showing dropout acts as an implicit gating mechanism.
result Dropout stabilizes a network's learning, allowing it to learn new tasks without forgetting old ones.
A simple gating mechanism improves deep learning convergence.
problem Vanishing or exploding gradients in deep networks.
method Introducing a zero-initialized parameter to each residual connection.
result Training deep networks (up to 120 layers) with fast convergence and better performance.
Study proposes GRU-D networks for missing value handling in road surface friction prediction.
problem Missing values in road surface friction data affect prediction accuracy.
method Gated Recurrent Unit (GRU) network with decay mechanism.
result GRU-D networks outperform baseline models in road surface friction prediction.