In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “Gated RNNs”
The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while…
Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mecha…
DeepESNs outperform ESN and GRUs in multivariate time-series prediction.
problem Comparing DeepESNs and gated RNNs for multivariate time-series prediction.
method Experimental comparison of DeepESNs and gated RNNs (Gated Recurrent Units, Long Short-Term Memory) on 4 polyphonic music tasks.
result DeepESNs outperform ESN and GRUs in terms of prediction accuracy and efficiency.
LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections. We present an alternative view to explain the success of LSTMs: the gates themselves are versatile recurrent models that provide more representational power than previously appreciated. We do…
FastGRNN improves RNN accuracy while drastically reducing model size.
problem Inaccurate training and inefficient prediction in RNNs.
method FastGRNN uses a residual connection and gate to achieve state-of-the-art accuracy with a much smaller model.
result FastGRNN achieves state-of-the-art accuracy with models up to 35x smaller than existing RNNs.
We present a novel recurrent neural network (RNN) based model that combines the remembering ability of unitary RNNs with the ability of gated RNNs to effectively forget redundant/irrelevant information in its memory. We achieve this by extending unitary RNNs with a gating mechanism. Our model is able to outperform LSTM…
Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent net…
New interpretation of RNN forget gate improves learnability for long-term sequential data.
problem Improving learnability of recurrent neural networks for long-term temporal dependencies.
method Generalized theory of gated RNNs, focusing on gradient behavior over time.
result Existing RNNs satisfy the gradient condition for initial training, suggesting validity of forget gate interpretation.
Gating units in GRUs and LSTMs create slow modes and control phase-space complexity.
problem Training challenges in RNNs due to exploding or vanishing gradients.
method Random matrix theory and mean-field theory applied to GRUs and LSTMs.
result Gates in GRUs and LSTMs lead to accumulation of slow modes and control phase-space complexity.
A new LSTM model reduces state updates and improves convergence for long sequences.
problem Vanishing gradient problem in RNNs and slow convergence on long sequences.
method Gaussian-gated LSTM (g-LSTM) with a time gate to control neuron updates.
result The g-LSTM model reduces state updates and computes by at least 10x compared to an equivalent LSTM.
Paper introduces a simpler gated RNN structure to better capture long-term dependencies.
problem Difficulty in learning long-term dependencies in RNNs.
method Proposes a grouped distributor unit (GDU) with partitioned hidden states and adaptive update rates.
result GDU outperforms LSTM and GRU on various tasks, including pathological and natural data.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
Complex numbers have long been favoured for digital signal processing, yet complex representations rarely appear in deep learning architectures. RNNs, widely used to process time series and sequence information, could greatly benefit from complex representations. We present a novel complex gated recurrent cell, which i…
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We…
Bayesian method sparsifies gated RNNs, improving speed and interpretability.
problem Sparsifying neural networks to reduce complexity and improve performance.
method Bayesian approach to sparsify weights, neurons, and gates in LSTM architectures.
result Sparsified gated RNNs speed up forward pass and improve compression.
Study compares MoE and RNN models for stock price prediction across volatility profiles.
problem Improving stock price prediction accuracy across different volatility levels.
method Dynamic Mixture of Experts model combining RNN and linear models, adjusting weights through a gating network.
result MoE model outperforms individual models in reducing prediction errors.
We sparsify gated RNNs by simplifying their structure.
problem Improving efficiency of RNNs by reducing their complexity.
method Adjust existing sparsification techniques to gated RNNs, sparsifying preactivations of gates.
result Simplified LSTM structure improves model performance and efficiency.
Study on recurrent neural networks' feature selection and memorization using F1B test.
problem Conflict between feature selection and memorization in sequence learning.
method Flagged-1-Bit (F1B) test, four recurrent network models studied analytically and experimentally.
result Conflict can be resolved by gating mechanism or increasing state dimension.
Recurrent Neural Networks (RNNs) with sophisticated units that implement a gating mechanism have emerged as powerful technique for modeling sequential signals such as speech or electroencephalography (EEG). The latter is the focus on this paper. A significant big data resource, known as the TUH EEG Corpus (TUEEG), has …
PF-RNNs use particle filtering to model uncertainty in RNNs for better sequential data prediction.
problem Highly variable and noisy sequential data.
method PF-RNNs maintain a latent state distribution as a set of particles, updating with Bayes rule.
result PF-RNNs outperform standard RNNs on various sequence prediction tasks.
Recurrent Neural Networks (RNNs) play a major role in the field of sequential learning, and have outperformed traditional algorithms on many benchmarks. Training deep RNNs still remains a challenge, and most of the state-of-the-art models are structured with a transition depth of 2-4 layers. Recurrent Highway Networks …
The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components. We propose a domain-s…
Study on RNNs' generalization bounds and variants.
problem Understanding the theoretical limits of RNNs and their variants.
method PAC-Learning framework, spectral norms, and total number of parameters.
result Significantly tighter generalization bound for vanilla RNNs.
Graph-structured data such as social networks, functional brain networks, gene regulatory networks, communications networks have brought the interest in generalizing deep learning techniques to graph domains. In this paper, we are interested to design neural networks for graphs with variable length in order to solve le…
Paper uses RNNs for more accurate indoor WiFi localization.
problem Accurate indoor WiFi localization using RSSI measurements.
method Proposes recurrent neural networks (RNNs) for trajectory positioning of RSSI data.
result Achieves an average localization error of 0.75 m with 80% under 1 m, outperforming conventional algorithms.
RNNs struggle with chaotic dynamics due to exploding gradients, but we found a way to optimize training.
problem Challenging training of RNNs with chaotic dynamics due to exploding gradients.
method Relating loss gradients to Lyapunov spectrum to optimize training on chaotic data.
result RNNs with chaotic dynamics always have diverging gradients, while stable ones have bounded gradients.
This research proposes a CL model for RNNs to handle sequential data without forgetting.
problem Learning in dynamic environments without forgetting previous knowledge for sequential data.
method A Recurrent Neural Network (RNN) model with Elastic Weight Consolidation (EWC) for CL.
result The proposed model outperforms EWC and RNNs on CL benchmarks for sequential data.
Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of encoder with attention that looks over the entire sequence and assigns a weight to each token independently. We present a mechanism for focusing…
This paper uses ODE to improve RNN models for time series data.
problem Improving RNN models for irregularly sampled time series data.
method Extending RNNs with Neural Ordinary Differential Equations (ODEs).
result New ODE-based RNN models reduce training and evaluation time.
ERNNs evolve hidden states on an ODE's equilibrium manifold to mitigate vanishing and exploding gradients.
problem Vanishing and exploding gradients in RNNs.
method Develop a novel family of RNNs (ERNNs) that evolve hidden states on the equilibrium manifold of an ODE.
result ERNNs achieve state-of-the-art accuracy with 3-10x speedups and 1.5-3x model size reduction.
Input-cell attention improves RNN saliency detection over time.
problem Vanishing saliency in RNNs makes them unreliable for detecting important features at arbitrary time intervals.
method Proposes input-cell attention, a novel RNN cell structure that uses a fixed-size matrix embedding to attend to different inputs from current or previous time steps.
result Input-cell attention produces a saliency map that can detect important features regardless of their occurrence in time.
Deep Recurrent Electricity Theft Detection in AMI Networks with Random Tuning of Hyper-parameterscs.LG
Modern smart grids rely on advanced metering infrastructure (AMI) networks for monitoring and billing purposes. However, such an approach suffers from electricity theft cyberattacks. Different from the existing research that utilizes shallow, static, and customer-specific-based electricity theft detectors, this paper p…
DIFFRNN verifies RNNs for safety and equivalence.
problem Certifying safety and equivalence of RNNs for resource-constrained platforms.
method Bounding nonlinear activation functions with linear constraints, solving constrained optimization problems, and using dReal SMT solver.
result DIFFRNN certifies the equivalence of optimized and original RNNs.
Study enhances neural network predictions for wave height using topological features.
problem Challenges in predicting wave heights due to short-term and long-term factors.
method Hybridization of persistent homology with neural networks for feature engineering.
result Significant improvements in score and reductions in errors for various neural network models.
Neural machine translation is a relatively new approach to statistical machine translation based purely on neural networks. The neural machine translation models often consist of an encoder and a decoder. The encoder extracts a fixed-length representation from a variable-length input sentence, and the decoder generates…
Study shows GRU model with dropout outperforms in Bitcoin price prediction.
problem Predicting Bitcoin price and volatility using machine learning.
method Advanced machine learning methods including GRU with recurrent dropout, feature engineering, and RMSE evaluation.
result Gated Recurrent Unit (GRU) model with recurrent dropout outperforms traditional models in Bitcoin price prediction.
Structured pruning method reduces RNN sizes and speeds up inference.
problem Large RNN models are hard to deploy on edge devices.
method Structured pruning through neuron selection, minimizing L0 norm of weight matrix.
result Nearly 20x speedup achieved without performance loss.
Deep neural networks improve forecasting of non-stationary time-series data.
problem Forecasting non-stationary time-series data with structural breaks and high volatility.
method Evaluation of DNN models including MLP, CNN, LSTM-RNN, and GRU-RNN on 10 Indian financial stocks.
result DNN models show better performance for single-step forecasting but degrade for multi-step forecasting, especially for long forecast periods.
Proposes POPQORN to quantify robustness of RNNs, including LSTMs and GRUs.
problem Vulnerability to adversarial attacks in recurrent neural networks.
method Develops POPQORN, a general algorithm to quantify robustness of RNNs.
result Demonstrates effectiveness on different network architectures and shows insights from robustness quantification on individual steps.
This study improves stock investment strategies using advanced neural networks.
problem Improving stock investment strategies for better performance.
method Used LSTM-GRU neural networks combined with SVM for stock prediction.
result LSTM-GRU outperformed benchmarks in stock predictions.
HCRNN uses hierarchical contexts to improve recommendation models.
problem Challenges in modeling user interest transitions and drifts in recommendation systems.
method Introduces HCRNN with three hierarchical contexts (global, local, temporary) and a hierarchical context-based gate structure.
result HCRNN outperformed other models in sequential recommendation tasks.
Paper addresses LSTM stability for thermal systems using infinity-norm.
problem Stability of LSTM networks in thermal systems.
method Derived ISS condition for LSTM, developed training strategy.
result ISS-promoted LSTM outperforms other models in thermal system case study.
Paper proposes a deep learning model to predict stock prices using sentiment analysis.
problem Predicting future stock movement using financial textual and numerical data.
method A blending ensemble deep learning model with two levels of RNNs, LSTM, and GRU followed by a fully connected neural network.
result The model improves prediction accuracy compared to traditional methods.
We present a new model, Predictive State Recurrent Neural Networks (PSRNNs), for filtering and prediction in dynamical systems. PSRNNs draw on insights from both Recurrent Neural Networks (RNNs) and Predictive State Representations (PSRs), and inherit advantages from both types of models. Like many successful RNN archi…
Relaxed Parameter Sharing: Effectively Modeling Time-Varying Relationships in Clinical Time-Seriescs.LG
Improved patient risk stratification with relaxed parameter sharing in clinical time-series data.
problem Learning time-varying relationships in clinical time-series data with limited training data.
method Proposed a novel RNN formulation based on a mixture model with relaxed parameter sharing over time.
result Relaxed parameter sharing leads to improved patient risk stratification performance in settings with limited data.
Gated Recurrent Unit (GRU) is a recently-developed variation of the long short-term memory (LSTM) unit, both of which are types of recurrent neural network (RNN). Through empirical evidence, both models have been proven to be effective in a wide variety of machine learning tasks such as natural language processing (Wen…