Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
Linear RC shows hierarchical temporal patterns in state signals.
problem Understanding hierarchical temporal representations in deep RNNs.
method Used linear recurrent units and frequency analysis on state signals.
result Linear RC reveals intrinsic hierarchical temporal structure.
New model explains complex, nonlinear systems with simpler units that switch based on observations.
problem Complex, nonlinear systems with switching dynamics.
method Recurrent switching linear dynamical systems (recSLDS) model.
result Models the switching behavior of simpler units based on observations or latent states.
RotRNN uses rotations to simplify long sequence modelling.
problem Complex initialisation and normalisation schemes in linear recurrent models.
method RotRNN employs rotation matrices to simplify and normalise linear recurrent models.
result RotRNN achieves competitive performance on long sequence modelling datasets.
Bayesian units improve speech recognition with minimal parameters.
problem Improving speech recognition models with fewer parameters.
method Derived Bayesian recurrent units integrated into deep learning frameworks.
result Adding Bayesian units improves speech recognition performance.
Linear recurrent networks explain reinforcement learning performance in partially observable settings.
problem Understanding why linear recurrent networks work in reinforcement learning with partial observability.
method Constructed and studied two linear filters for HMMs and action-controlled HMMs.
result Linear filters serve as sufficient statistics and reduce state ambiguity, explaining empirical reinforcement learning success.
CRUs model irregular time series with continuous hidden states.
problem Handling irregular time intervals in sequential data.
method Continuous Recurrent Units (CRUs) that integrate hidden states via a linear stochastic differential equation.
result CRUs outperform methods based on neural ordinary differential equations in irregular time series interpolation.
ARU adapts deep forecasting models in streaming data with efficient updates.
problem Adapting deep globally trained models for streaming data efficiently.
method ARU combines deep global models with closed-form linear models for per-series adaptation.
result ARU outperforms local adaptation methods on various datasets.
Theoretical analysis shows RNNs with various nonlinearities benefit from depth efficiency.
problem Theoretical understanding of RNNs' efficiency is limited.
method Extended analysis to RNNs with Rectified Linear Unit (ReLU) and other nonlinearities.
result Various nonlinear RNNs also benefit from depth efficiency.
Analyzes convex structures in Teichmüller space unit tangent spheres.
problem Characterize faces and extreme points of unit tangent spheres in Teichmüller space.
method Analyzes Finsler infinitesimal balls of Thurston metric, characterizes faces, exposed faces, and extreme points.
result Characterizes faces and extreme points of unit tangent spheres in Teichmüller space.
SRU learns long-term dependencies without gates, outperforming LSTMs and GRUs.
problem Learning long-term dependencies in data efficiently.
method Developed an un-gated statistical recurrent unit (SRU) that keeps moving averages of statistics.
result SRU outperforms LSTMs and GRUs in various tasks, often outperforming both.
Convolutional LSTM networks outperform GRU in EEG seizure detection.
problem Seizure detection in EEG signals.
method Comparison of LSTM and GRU units, hybrid CNN-RNN architecture, various initialization and regularization methods.
result Convolutional LSTM networks achieve 30% sensitivity at 6 false alarms per 24 hours.
Improved GRU model with weighted time-delay feedback for long-term dependencies.
problem Modeling long-term dependencies in sequential data.
method Introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism.
result τ-GRU outperforms state-of-the-art models on various tasks.
Three MGU variants reduce parameters and improve recurrent neural network performance.
problem Improving recurrent neural network performance with fewer parameters.
method Introducing three MGU variants with simplified forget-gate dynamic equations.
result MGU2 variant outperformed MGU on MNIST and RNT datasets.
Three GRU variants reduce parameters in RNNs, improving efficiency.
problem Reducing computational expense in RNNs.
method Three variants of GRU with reduced parameters in update and reset gates.
result Variant models perform similarly to original GRU RNN models.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
Paper proposes dual recurrent attention units for VQA models.
problem Comprehending visual and textual data for accurate question answering.
method Introduces and evaluates recurrent attention mechanisms in VQA models.
result Dual Recurrent Attention Units (RAUs) improve VQA performance.
Traditional convolutional layers extract features from patches of data by applying a non-linearity on an affine function of the input. We propose a model that enhances this feature extraction process for the case of sequential data, by feeding patches of the data into a recurrent neural network and using the outputs or…
RNNs can store information in parameters and input history, with capacity and trainability being key factors.
problem Understanding and optimizing the capacity and trainability of RNNs.
method Experimental analysis of various RNN architectures, including comparison of training difficulty and per-task parameter capacity.
result RNNs can store an amount of task information linear in the number of parameters (approximately 5 bits per parameter) and can store one real number from input history per hidden unit.
New eGRU unit improves keyword spotting on ultra-low-power devices.
problem Resource constraints of edge devices for neural network deployment.
method Optimized recurrent unit architecture for ultra-low power.
result eGRU is 60x faster and 10x smaller than GRU, maintaining accuracy.
New RNN units vary computation to match data flow, improving performance.
problem Fixed computation in RNNs limits model capacity and speed for variable data.
method Modified RNN units learn to vary computation per step.
result Variable computation leads to better performance and fewer operations.
New method improves learning of long-term dependencies in recurrent models.
problem Improving learning of long-term dependencies in recurrent neural networks.
method Proves learnable gates in recurrent models provide quasi-invariance to time transformations and recovers part of LSTM architecture from axiomatic approach.
result New chrono initialization of gate biases greatly improves learning of long-term dependencies.
Paper presents RGNN for better graph node representation learning.
problem Node representation learning with graph neural networks.
method Recurrent Graph Neural Network (RGNN) with recurrent units.
result RGNN achieves state-of-the-art results on three benchmarks.
New LTC RNNs can approximate any continuous system with fewer units.
problem Approximating continuous dynamical systems with neural networks.
method Introducing LTC RNNs with variable time-constant synaptic transmission.
result LTC RNNs can approximate any n-dimensional continuous dynamical system. Neural network combines GRU and SVM for better intrusion detection.
problem Improving accuracy in binary classification for network intrusion detection.
method Integrates GRU with SVM as final output layer, replacing Softmax and cross-entropy.
result GRU-SVM model outperforms conventional GRU-Softmax model in accuracy and prediction time.
ENRNN uses eigenvalue normalization for short-term memory in RNNs.
problem Vanishing/exploding gradient problem and long-term dependency modeling.
method Eigenvalue normalization of recurrent matrix to simulate short-term memory.
result ENRNN outperforms existing RNN variants in experiments.
This paper improves speech synthesis using a DGP with SRU for naturalness.
problem Improving naturalness in synthetic speech.
method Deep Gaussian process with a recurrent architecture using SRU.
result SRU-DGP outperforms other models in naturalness of synthetic speech.
Combines gating and tensor products for RNNs to improve performance.
problem Improving RNNs' ability to capture long-term dependencies.
method Proposes a novel RNN architecture combining gating mechanism and tensor products.
result Significant performance improvement on word-level and character-level language modeling tasks.
Bayesian approach improves neural network recurrence.
problem Improving neural network recurrence mechanisms.
method Introducing Bayesian recurrence relations and gates.
result Bayesian approach can perform as well as or better than conventional recurrent networks.
Paper analyzes convergence of SGD in RNNs, proving linear convergence rate.
problem Analyzing convergence of SGD in multi-layer recurrent neural networks.
method Developed tools to analyze multi-layer networks with ReLU activations.
result SGD achieves linear convergence rate in training RNNs with sufficient neurons.
High-order RNNs reduce speech recognition errors.
problem Vanishing gradients in RNNs.
method High-order RNNs with multiple connections from previous time steps.
result HORNNs reduce WER by 4.2% and 6.3% over RNNs.
Study shows GRU model with dropout outperforms in Bitcoin price prediction.
problem Predicting Bitcoin price and volatility using machine learning.
method Advanced machine learning methods including GRU with recurrent dropout, feature engineering, and RMSE evaluation.
result Gated Recurrent Unit (GRU) model with recurrent dropout outperforms traditional models in Bitcoin price prediction.
Paper proposes a hybrid MTL framework for improved stock market prediction accuracy.
problem Inaccurate stock market predictions due to financial data's complexities.
method Multi-layer hybrid MTL structure with Transformer, BiGRU, and KAN.
result Achieved low MAE (1.078), MAPE (0.012), and high R^2 (0.98) compared to other models.
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
Machine learning predicts movie genres from summaries with high accuracy.
problem Predicting movie genres from plot summaries.
method Used Naive Bayes, Word2Vec+XGBoost, Recurrent Neural Networks, and Gated Recurrent Units (GRU) for text classification and multi-label problem.
result GRU neural networks achieve the best result with a Jaccard Index of 50.0%, F-score of 0.56, and hit rate of 80.5%.
Inspired by LSTMs, a new neural network model mimics cortical microcircuits.
problem Understanding the computational principles of cortical microcircuits.
method Introducing a gated-recurrent neural network (subLSTM) based on inhibitory cells.
result SubLSTM units achieve similar performance to LSTM units in sequential tasks.
Proposes FRU to stabilize gradients and improve long-term dependencies in RNNs.
problem Challenges in training RNNs for tasks with long-term dependencies.
method Introduces Fourier Recurrent Units (FRU) that stabilize gradients and improve expressivity.
result FRU stabilizes gradients and has stronger expressive power, leading to better performance.
GRUs exhibit diverse dynamical behaviors but cannot mimic continuous attractors.
problem Understanding and predicting the dynamics of GRUs for neural data.
method Continuous time dynamical systems analysis of GRU networks.
result GRUs can represent stable limit cycles, multi-stable dynamics, and homoclinic bifurcations but not continuous attractors.
We prove quantitative recurrence and large deviations results for the Teichmuller geodesci flow on a connected component of a stratum of the moduli space Qg of holomorphic unit-area quadratic differentials on a compact genus g≥2 surface.
GORU combines unitary and gated RNNs for better long-term memory management.
problem Learning to effectively manage long-term memory in neural networks.
method Extending unitary RNNs with a gating mechanism to forget irrelevant information.
result GORU outperforms LSTMs, GRUs, and Unitary RNNs on long-term dependency tasks.
Improved video prediction with bijective Gated Recurrent Units.
problem Ill-posed future video prediction with high variability and error propagation.
method Introduces bijective Gated Recurrent Units for state sharing in auto-encoders.
result Significant reduction in computational cost and memory usage compared to state-of-the-art approaches.
Neural network predicts falls in elderly people up to 10 minutes in advance.
problem Falls prevention in elderly people, especially in aging societies.
method Gated Recurrent Unit (GRU) based neural networks model using heart rate and mean blood pressure signals.
result Predicted syncope occurrence approximately 10 minutes before manual markers.
New method estimates uncertainty in GRUs without sampling.
problem Uncertainty estimation in deep learning models.
method Exponential families for deterministic uncertainty quantification.
result Sampling-free uncertainty estimation for GRUs.
Efficient deep GNNs achieve state-of-the-art performance without training.
problem Efficiency issue in deep graph neural networks.
method Representing graphs as fixed points of dynamical systems and using small, sparse, untrained recurrent networks.
result Small deep GNNs without training can achieve or improve state-of-the-art performance.
DeepProteomics uses neural networks to classify protein families efficiently.
problem Lack of functional annotation for many protein sequences in databases.
method Used RNN, LSTM, GRU, and deep neural network models on a dataset of 40,433 proteins.
result Achieved maximum 78% accuracy in classifying protein families.
Neural networks predict traffic flow in smart cities.
problem Forecasting stochastic and nonlinear traffic flow.
method Various recurrent neural networks trained on intersection data.
result Vector output model with gated recurrent units performed best.
In this paper we propose and investigate a novel nonlinear unit, called Lp unit, for deep neural networks. The proposed Lp unit receives signals from several projections of a subset of units in the layer below and computes a normalized Lp norm. We notice two interesting interpretations of the Lp unit. First…