Deep network improves NILM by distinguishing on/off states of appliances.
problem Break down household aggregate electricity consumption into individual appliance usages.
method Subtask gated network combining regression and classification subtasks.
result Surpasses state-of-the-art performance for most benchmark cases.
Agent learns to solve unseen tasks with subtask dependencies.
problem Generalizing to unseen tasks with subtask dependencies.
method Neural subtask graph solver (NSGS) with graph reward propagation for pre-training and actor-critic finetuning.
result Agent can find near-optimal subtask execution and generalize to unseen subtask graphs.
In this paper we combine one method for hierarchical reinforcement learning - the options framework - with deep Q-networks (DQNs) through the use of different "option heads" on the policy network, and a supervisory network for choosing between the different options. We utilise our setup to investigate the effects of ar…
Task focuses on fact checking in Q&A forums, improving over baseline systems.
problem Fact checking in community Q&A forums to distinguish factual from opinion.
method Two subtasks: distinguishing factual vs. opinion/advice/socializing, predicting answer truthfulness.
result Improved over baseline systems for both subtasks, but not for Subtask B.
RoBERTa model detects counterfactual statements in text.
problem Detecting and extracting counterfactual statements from text.
method Used RoBERTa language representation model for both subtasks.
result RoBERTa achieved top performance in both subtasks at SemEval-2020.
Meta-learner infers subtask graph to adapt quickly to unknown tasks.
problem Adapting to unknown tasks with unknown subtask dependencies in few-shot RL.
method Meta-learner with Subtask Graph Inference (MSGI) and UCB-inspired intrinsic reward.
result Accurately infers latent task parameters and adapts more efficiently.
MAML optimizes shared priors for subtasks in a nonconvex meta-objective.
problem Understanding global optimality of MAML for nonconvex meta-objectives.
method Characterizes optimality gap of MAML stationary points via first-order optimization methods.
result Establishes global optimality of MAML for both RL and supervised learning.
Bayesian method sparsifies gated RNNs, improving speed and interpretability.
problem Sparsifying neural networks to reduce complexity and improve performance.
method Bayesian approach to sparsify weights, neurons, and gates in LSTM architectures.
result Sparsified gated RNNs speed up forward pass and improve compression.
HAL learns hierarchical affordances to prune impossible subtasks, improving reinforcement learning efficiency.
problem Reinforcement learning struggles with complex hierarchical dependency structures.
method HAL learns a model of hierarchical affordances to prune impossible subtasks.
result HAL agents are better at learning complex tasks, navigating stochastic environments, and acquiring diverse skills.
We sparsify gated RNNs by simplifying their structure.
problem Improving efficiency of RNNs by reducing their complexity.
method Adjust existing sparsification techniques to gated RNNs, sparsifying preactivations of gates.
result Simplified LSTM structure improves model performance and efficiency.
GLCB uses Gated Linear Networks for online contextual bandits.
problem Online learning in contextual bandits with uncertainty estimation.
method Gated Linear Networks (GLNs) for prediction and uncertainty estimation.
result GLCB outperforms state-of-the-art methods in online contextual bandits.
Improved RNNs with flexible gates using kernel activation functions.
problem Modeling long-term dependencies in sequential data.
method Designed a more flexible architecture with adaptable parameters using kernel activation functions.
result Improved accuracy with negligible computational cost and speed-up in training iterations.
New algorithm trains neural nets on simple skills to learn complex tasks faster.
problem Learning complex tasks through simple imitation.
method Train neural networks on simple, easy-to-learn skills to accelerate learning of complex, hard-to-learn tasks.
result Consistently outperforms state-of-the-art baseline in training speed and performance.
This paper analyzes how gradient descent can learn parameters in mixture-of-experts models with gating.
problem Understanding parameter recovery in mixture-of-experts models with gating.
method Careful analysis of the optimization landscape and design of two distinct loss functions.
result Gradient descent can accurately learn parameters in mixture-of-experts models with gating.
The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while…
Improved neural networks with conditional gating and reduced computational cost.
problem Training large capacity neural networks with high accuracy and low computational cost.
method Fine-grained channel gating using batch-shaping and residual blocks.
result Conditional gating improves network accuracy and computational efficiency.
A major contributing factor to the recent advances in deep neural networks is structural units that let sensory information and gradients to propagate easily. Gating is one such structure that acts as a flow control. Gates are employed in many recent state-of-the-art recurrent models such as LSTM and GRU, and feedforwa…
A new LSTM model reduces state updates and improves convergence for long sequences.
problem Vanishing gradient problem in RNNs and slow convergence on long sequences.
method Gaussian-gated LSTM (g-LSTM) with a time gate to control neuron updates.
result The g-LSTM model reduces state updates and computes by at least 10x compared to an equivalent LSTM.
The paper analyzes the role of ReLU gates in deep learning networks.
problem Understanding the role of gates in deep learning networks.
method Developed neural path features (NPF) and neural path values (NPV) to characterize the active sub-networks during training.
result The neural path kernel associated with NPFs is a fundamental quantity that characterizes the information stored in the gates of a DNN.
Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-ini…
Deep gated networks help understand training and generalization in deep learning.
problem Understanding the role of SGD in training and generalization of deep neural networks with ReLU activation.
method Developed deep gated networks (DGNs) as a framework to analyze training and generalization in DNNs with ReLU activation.
result Gate adaptation is key for generalization in deep neural networks.
A trainable gate optimizes neural network selection and pruning.
problem Optimizing neural networks for specific tasks.
method Introduces a trainable gate function to make discrete selection problems differentiable.
result Efficiently optimizes arbitrary neural networks across various tasks.
Over 50 million scholarly articles have been published: they constitute a unique repository of knowledge. In particular, one may infer from them relations between scientific concepts, such as synonyms and hyponyms. Artificial neural networks have been recently explored for relation extraction. In this work, we continue…
This paper introduces channel gating, a dynamic, fine-grained, and hardware-efficient pruning scheme to reduce the computation cost for convolutional neural networks (CNNs). Channel gating identifies regions in the features that contribute less to the classification result, and skips the computation on a subset of the …
New Gated Linear Unit (GaLU) improves neural network performance.
problem Theoretical gap between ReLU networks and empirical observations in shallow networks.
method Introduced Gated Linear Unit (GaLU) by decoupling linearity from gating.
result Improved memorization and generalization results for GaLU networks.
Gated Path Planning Network improves path planning efficiency and stability.
problem Training instability and optimization issues in Value Iteration Networks.
method Reframed VINs as recurrent-convolutional networks with gated update equations.
result Gated Path Planning Network outperforms VINs in various metrics.
ExGate improves neural network accuracy with feature-based attention.
problem Efficiency and accuracy in artificial neural networks for feature-based attention.
method Externally controlled neuron gating in artificial neural networks.
result 5% increase in classification accuracy on CIFAR-10 dataset.
A new method quantizes LSTM gate parameters without performance loss.
problem Quantization loss in LSTM gate parameters without performance degradation.
method Lossy quantization of gate parameters during training, weight parameters adjust to offset quantization loss.
result F1 score decreased by only 0.7% on Named Entity Recognition dataset.
A new method for differentiable structured sparsity improves neural network performance and sparsity.
problem Non-differentiability of structured sparsity penalties in neural networks.
method Introducing D-Gating, a differentiable approach to structured overparameterization. result The D-Gating objective converges to the L2,2/D-regularized loss and induces sparse learning dynamics. Study reveals biases in gradient descent for GLNs, improving neural network performance.
problem Understanding and improving the inductive biases of deep neural networks.
method Derive infinite-time training limit of gated linear networks and generalize to other networks.
result Theoretical framework captures key inductive biases of ReLU networks.
This paper proposes a semi-tied unit approach to improve efficiency in LSTM and highway networks.
problem Efficiency issues in gating mechanisms of LSTM and highway networks.
method Semi-tied units (STUs) using one shared weight matrix with extra scaling factors.
result Reduced calculation and storage costs by a factor of three for highway networks and four for LSTMs.
Gradient Gating improves deep GNNs by modulating message passing updates.
problem Oversmoothing and performance degradation in deep GNNs.
method Gradient gating mechanism for multi-rate message passing.
result G2 framework alleviates oversmoothing and achieves state-of-the-art performance. Monotone neural networks can approximate and interpolate functions efficiently.
problem Understanding the efficiency and expressiveness of monotone neural networks.
method Solving the monotone interpolation problem using depth-4 networks and comparing size bounds with arbitrary networks.
result Monotone neural networks can approximate and interpolate functions efficiently, but may require exponential size in high dimensions.
Paper interprets ResNets via gate-network controls and deep-layer classifications.
problem Understanding the performance mechanism of ResNets.
method Constructs typical solutions using gate-network controls and deep-layer classifications.
result Proves the universal-approximation capability of ResNets.
Improved GNN handles long-range dependencies in multi-relational graphs.
problem Vanishing gradients in GNNs for multi-relational graphs.
method Proposes a Gated Graph Neural Network with improved long-range dependency handling.
result Outperforms popular GNN models in synthetic tasks.
Recurrent neural networks with various types of hidden units have been used to solve a diverse range of problems involving sequence data. Two of the most recent proposals, gated recurrent units (GRU) and minimal gated units (MGU), have shown comparable promising results on example public datasets. In this paper, we int…
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
Methods from convex optimization are widely used as building blocks for deep learning algorithms. However, the reasons for their empirical success are unclear, since modern convolutional networks (convnets), incorporating rectifier units and max-pooling, are neither smooth nor convex. Standard guarantees therefore do n…
Gated Linear Networks bypass feature learning for fast online learning.
problem Fast online learning and feature learning trade-offs in neural networks.
method Distributed and local credit assignment mechanism, data-dependent gating, online convex optimization.
result GLNs achieve universal learning capabilities and resilience to catastrophic forgetting.
GCRNNs improve graph problem solving with fewer parameters.
problem Graph process problems like earthquake epicenter identification and weather prediction.
method GCRNNs use convolutional filter banks and time-gated variations of GCRNNs (Gated GCRNNs) to improve performance.
result GCRNNs significantly improve performance over GNNs and another graph recurrent architecture.
Gating units in GRUs and LSTMs create slow modes and control phase-space complexity.
problem Training challenges in RNNs due to exploding or vanishing gradients.
method Random matrix theory and mean-field theory applied to GRUs and LSTMs.
result Gates in GRUs and LSTMs lead to accumulation of slow modes and control phase-space complexity.
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
For any positive integer k, there exist neural networks with Θ(k3) layers, Θ(1) nodes per layer, and Θ(1) distinct parameters which can not be approximated by networks with O(k) layers unless they are exponentially large --- they must possess Ω(2k) nodes. This result is proved here for a class o…
Deep neural network learns meaningful factors to predict stock returns.
problem Predicting excess returns of assets like Tesla stock.
method 5-layer deep neural network with gated activation layer to filter noise.
result Proposed model outperforms in predicting stock returns over 2,000 stocks.
Recurrent Neural Networks (RNNs) with sophisticated units that implement a gating mechanism have emerged as powerful technique for modeling sequential signals such as speech or electroencephalography (EEG). The latter is the focus on this paper. A significant big data resource, known as the TUH EEG Corpus (TUEEG), has …
Theory explains how recurrent networks remember sequences.
problem Understanding how recurrent networks remember sequences and perform well.
method Mean field theory and random matrix theory applied to RNNs with gating mechanisms.
result Gated RNNs outperform non-gated RNNs in remembering sequences.
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
FleXOR trains fractional quantization for neural networks, improving accuracy and size.
problem Quantization limits to integer bits restricts compression and accuracy.
method Encryption algorithm with XOR gates for fractional bits during inference.
result FleXOR achieves high accuracy with fractional sub-1-bit weights.