SRU learns long-term dependencies without gates, outperforming LSTMs and GRUs.
problem Learning long-term dependencies in data efficiently.
method Developed an un-gated statistical recurrent unit (SRU) that keeps moving averages of statistics.
result SRU outperforms LSTMs and GRUs in various tasks, often outperforming both.
New RNN units vary computation to match data flow, improving performance.
problem Fixed computation in RNNs limits model capacity and speed for variable data.
method Modified RNN units learn to vary computation per step.
result Variable computation leads to better performance and fewer operations.
Bayesian units improve speech recognition with minimal parameters.
problem Improving speech recognition models with fewer parameters.
method Derived Bayesian recurrent units integrated into deep learning frameworks.
result Adding Bayesian units improves speech recognition performance.
New method infers nonlinear Granger causality from time series data.
problem Inferring nonlinear Granger causality from time series data.
method Statistical Recurrent Units (SRUs) for modeling nonlinear interactions.
result The proposed economy-SRU model outperforms existing models in inferring Granger causality.
Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
Convolutional LSTM networks outperform GRU in EEG seizure detection.
problem Seizure detection in EEG signals.
method Comparison of LSTM and GRU units, hybrid CNN-RNN architecture, various initialization and regularization methods.
result Convolutional LSTM networks achieve 30% sensitivity at 6 false alarms per 24 hours.
ARU adapts deep forecasting models in streaming data with efficient updates.
problem Adapting deep globally trained models for streaming data efficiently.
method ARU combines deep global models with closed-form linear models for per-series adaptation.
result ARU outperforms local adaptation methods on various datasets.
Improved GRU model with weighted time-delay feedback for long-term dependencies.
problem Modeling long-term dependencies in sequential data.
method Introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism.
result τ-GRU outperforms state-of-the-art models on various tasks.
Three MGU variants reduce parameters and improve recurrent neural network performance.
problem Improving recurrent neural network performance with fewer parameters.
method Introducing three MGU variants with simplified forget-gate dynamic equations.
result MGU2 variant outperformed MGU on MNIST and RNT datasets.
Three GRU variants reduce parameters in RNNs, improving efficiency.
problem Reducing computational expense in RNNs.
method Three variants of GRU with reduced parameters in update and reset gates.
result Variant models perform similarly to original GRU RNN models.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
ParaRNN improves RNN interpretability and parallelizability for time-dependent data.
problem Limited interpretability and slow training of RNNs.
method Parallelized RNN with additive representation and recurrence features.
result ParaRNN achieves comparable performance to vanilla RNNs but with improved interpretability and efficiency.
A new memory-efficient sign language translation model reduces weight usage.
problem Memory constraints in real-time sign language translation.
method Variational Bayesian sequence-to-sequence network with Gaussian posterior and Indian Buffet Process prior.
result The proposed model achieves substantial weight compression without compromising performance.
Improves session-based recommendation by modeling temporal dynamics.
problem Data sparsity in session-based recommendation.
method Recurrent Latent Variable Networks (RLVN) with variational inference.
result Effective in scaling to large real-world datasets and improving recommendation accuracy.
Paper proposes dual recurrent attention units for VQA models.
problem Comprehending visual and textual data for accurate question answering.
method Introduces and evaluates recurrent attention mechanisms in VQA models.
result Dual Recurrent Attention Units (RAUs) improve VQA performance.
Linear recurrent networks explain reinforcement learning performance in partially observable settings.
problem Understanding why linear recurrent networks work in reinforcement learning with partial observability.
method Constructed and studied two linear filters for HMMs and action-controlled HMMs.
result Linear filters serve as sufficient statistics and reduce state ambiguity, explaining empirical reinforcement learning success.
New eGRU unit improves keyword spotting on ultra-low-power devices.
problem Resource constraints of edge devices for neural network deployment.
method Optimized recurrent unit architecture for ultra-low power.
result eGRU is 60x faster and 10x smaller than GRU, maintaining accuracy.
New method improves learning of long-term dependencies in recurrent models.
problem Improving learning of long-term dependencies in recurrent neural networks.
method Proves learnable gates in recurrent models provide quasi-invariance to time transformations and recovers part of LSTM architecture from axiomatic approach.
result New chrono initialization of gate biases greatly improves learning of long-term dependencies.
Paper presents RGNN for better graph node representation learning.
problem Node representation learning with graph neural networks.
method Recurrent Graph Neural Network (RGNN) with recurrent units.
result RGNN achieves state-of-the-art results on three benchmarks.
New LTC RNNs can approximate any continuous system with fewer units.
problem Approximating continuous dynamical systems with neural networks.
method Introducing LTC RNNs with variable time-constant synaptic transmission.
result LTC RNNs can approximate any n-dimensional continuous dynamical system. CRUs model irregular time series with continuous hidden states.
problem Handling irregular time intervals in sequential data.
method Continuous Recurrent Units (CRUs) that integrate hidden states via a linear stochastic differential equation.
result CRUs outperform methods based on neural ordinary differential equations in irregular time series interpolation.
ENRNN uses eigenvalue normalization for short-term memory in RNNs.
problem Vanishing/exploding gradient problem and long-term dependency modeling.
method Eigenvalue normalization of recurrent matrix to simulate short-term memory.
result ENRNN outperforms existing RNN variants in experiments.
This paper improves speech synthesis using a DGP with SRU for naturalness.
problem Improving naturalness in synthetic speech.
method Deep Gaussian process with a recurrent architecture using SRU.
result SRU-DGP outperforms other models in naturalness of synthetic speech.
Combines gating and tensor products for RNNs to improve performance.
problem Improving RNNs' ability to capture long-term dependencies.
method Proposes a novel RNN architecture combining gating mechanism and tensor products.
result Significant performance improvement on word-level and character-level language modeling tasks.
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image…
Bayesian approach improves neural network recurrence.
problem Improving neural network recurrence mechanisms.
method Introducing Bayesian recurrence relations and gates.
result Bayesian approach can perform as well as or better than conventional recurrent networks.
Study shows GRU model with dropout outperforms in Bitcoin price prediction.
problem Predicting Bitcoin price and volatility using machine learning.
method Advanced machine learning methods including GRU with recurrent dropout, feature engineering, and RMSE evaluation.
result Gated Recurrent Unit (GRU) model with recurrent dropout outperforms traditional models in Bitcoin price prediction.
Analyzes convex structures in Teichmüller space unit tangent spheres.
problem Characterize faces and extreme points of unit tangent spheres in Teichmüller space.
method Analyzes Finsler infinitesimal balls of Thurston metric, characterizes faces, exposed faces, and extreme points.
result Characterizes faces and extreme points of unit tangent spheres in Teichmüller space.
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
Machine learning predicts movie genres from summaries with high accuracy.
problem Predicting movie genres from plot summaries.
method Used Naive Bayes, Word2Vec+XGBoost, Recurrent Neural Networks, and Gated Recurrent Units (GRU) for text classification and multi-label problem.
result GRU neural networks achieve the best result with a Jaccard Index of 50.0%, F-score of 0.56, and hit rate of 80.5%.
Linear RC shows hierarchical temporal patterns in state signals.
problem Understanding hierarchical temporal representations in deep RNNs.
method Used linear recurrent units and frequency analysis on state signals.
result Linear RC reveals intrinsic hierarchical temporal structure.
CTRNNs improve blood glucose forecasting in ICU, outperforming traditional models.
problem Forecasting blood glucose in ICU with irregular measurements.
method Continuous time autoregressive recurrent neural networks (CTRNNs) using neural ODE or neural flow layers.
result CTRNNs generally outperform traditional autoregressive models in probabilistic forecasting of blood glucose.
Inspired by LSTMs, a new neural network model mimics cortical microcircuits.
problem Understanding the computational principles of cortical microcircuits.
method Introducing a gated-recurrent neural network (subLSTM) based on inhibitory cells.
result SubLSTM units achieve similar performance to LSTM units in sequential tasks.
Proposes FRU to stabilize gradients and improve long-term dependencies in RNNs.
problem Challenges in training RNNs for tasks with long-term dependencies.
method Introduces Fourier Recurrent Units (FRU) that stabilize gradients and improve expressivity.
result FRU stabilizes gradients and has stronger expressive power, leading to better performance.
GRUs exhibit diverse dynamical behaviors but cannot mimic continuous attractors.
problem Understanding and predicting the dynamics of GRUs for neural data.
method Continuous time dynamical systems analysis of GRU networks.
result GRUs can represent stable limit cycles, multi-stable dynamics, and homoclinic bifurcations but not continuous attractors.
We prove quantitative recurrence and large deviations results for the Teichmuller geodesci flow on a connected component of a stratum of the moduli space Qg of holomorphic unit-area quadratic differentials on a compact genus g≥2 surface.
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
Neural networks predict traffic flow in smart cities.
problem Forecasting stochastic and nonlinear traffic flow.
method Various recurrent neural networks trained on intersection data.
result Vector output model with gated recurrent units performed best.
GORU combines unitary and gated RNNs for better long-term memory management.
problem Learning to effectively manage long-term memory in neural networks.
method Extending unitary RNNs with a gating mechanism to forget irrelevant information.
result GORU outperforms LSTMs, GRUs, and Unitary RNNs on long-term dependency tasks.
Improved video prediction with bijective Gated Recurrent Units.
problem Ill-posed future video prediction with high variability and error propagation.
method Introduces bijective Gated Recurrent Units for state sharing in auto-encoders.
result Significant reduction in computational cost and memory usage compared to state-of-the-art approaches.
Neural network predicts falls in elderly people up to 10 minutes in advance.
problem Falls prevention in elderly people, especially in aging societies.
method Gated Recurrent Unit (GRU) based neural networks model using heart rate and mean blood pressure signals.
result Predicted syncope occurrence approximately 10 minutes before manual markers.
New method estimates uncertainty in GRUs without sampling.
problem Uncertainty estimation in deep learning models.
method Exponential families for deterministic uncertainty quantification.
result Sampling-free uncertainty estimation for GRUs.
Efficient deep GNNs achieve state-of-the-art performance without training.
problem Efficiency issue in deep graph neural networks.
method Representing graphs as fixed points of dynamical systems and using small, sparse, untrained recurrent networks.
result Small deep GNNs without training can achieve or improve state-of-the-art performance.
New model explains complex, nonlinear systems with simpler units that switch based on observations.
problem Complex, nonlinear systems with switching dynamics.
method Recurrent switching linear dynamical systems (recSLDS) model.
result Models the switching behavior of simpler units based on observations or latent states.
DeepProteomics uses neural networks to classify protein families efficiently.
problem Lack of functional annotation for many protein sequences in databases.
method Used RNN, LSTM, GRU, and deep neural network models on a dataset of 40,433 proteins.
result Achieved maximum 78% accuracy in classifying protein families.
In this paper we propose and investigate a novel nonlinear unit, called Lp unit, for deep neural networks. The proposed Lp unit receives signals from several projections of a subset of units in the layer below and computes a normalized Lp norm. We notice two interesting interpretations of the Lp unit. First…
Neural network predicts cardiovascular events from EHRs with high accuracy.
problem Predicting onset of cardiovascular diseases from electronic health records.
method Multi-task gated recurrent units with attention mechanism.
result Model outperforms clinical risk scores in predicting stroke and myocardial infarction.