mGRN improves multivariate time series prediction by managing marginal and joint memories.
problem Extracting dependencies in multivariate sequential data with strong serial and cross-sectional dependencies.
method Developed a novel recurrent network architecture, Memory-Gated Recurrent Networks (mGRN), with gates for marginal and joint memories.
result mGRN consistently outperforms state-of-the-art architectures on various public datasets.
Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.
problem Understanding memory and forecasting capabilities in networks with dependent inputs.
method Formulated bounds for memory and forecasting capacities in terms of network size and input properties.
result Proved that memory capacity for linear recurrent networks with independent inputs is given by the rank of the controllability matrix.
Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation b…
A new memory-efficient sign language translation model reduces weight usage.
problem Memory constraints in real-time sign language translation.
method Variational Bayesian sequence-to-sequence network with Gaussian posterior and Indian Buffet Process prior.
result The proposed model achieves substantial weight compression without compromising performance.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
ARMIN improves memory efficiency and lightness in neural networks.
problem Complex memory addressing and inefficient memory information exploitation in previous MANNs.
method ARMIN uses hidden state for automatic memory addressing and a novel RNN cell for memory integration.
result ARMIN achieves lower computational overhead and similar performances compared to vanilla LSTM.
ENRNN uses eigenvalue normalization for short-term memory in RNNs.
problem Vanishing/exploding gradient problem and long-term dependency modeling.
method Eigenvalue normalization of recurrent matrix to simulate short-term memory.
result ENRNN outperforms existing RNN variants in experiments.
Improved recurrent neural networks learn long-term dependencies through multi-scale memory.
problem Capturing long-term dependencies in recurrent neural networks.
method Incremental training of a modular RNN architecture with multi-scale hidden states.
result Incremental training and multi-scale memory enhance RNNs' ability to learn long-term dependencies.
QRPNNs use quaternion-valued recurrent correlation neural networks to solve cross-talk issues.
problem Cross-talk problem in QRCNNs.
method Combining non-local projection learning with QRCNNs.
result QRPNNs exhibit greater storage capacity and noise tolerance.
Extended LSTMs improve volatility prediction by 20%.
problem Predicting asset price volatility with long memory.
method Extended LSTMs with multiple flexible timescales.
result Extended LSTMs outperform rough volatility predictions by 20%.
RNN beats Lee-Carter in forecasting mortality rates.
problem Forecasting mortality rates across different demographics.
method Long Short-Term Memory (LSTM) recurrent neural network trained on multiple countries, ages, and sexes.
result RNN model outperforms the Lee-Carter model in mortality rate forecasting.
The memory capacity of linear echo state networks is accurately calculated using new numerical methods.
problem Numerical evaluations of memory capacity in recurrent neural networks often contradict theoretical bounds.
method Developed robust numerical approaches exploiting MC neutrality with respect to the input mask matrix.
result Memory curves fully agree with theory when using the proposed methods.
New kernel-based models improve on traditional neural methods in sequence modeling.
problem Sequence modeling challenges in natural language processing and neuroscience.
method Kernel-based recurrent neural networks and convolutional neural networks.
result Kernel-based models perform on par or better than traditional neural methods.
Recurrent neural networks (RNN) are at the core of modern automatic speech recognition (ASR) systems. In particular, long-short term memory (LSTM) recurrent neural networks have achieved state-of-the-art results in many speech recognition tasks, due to their efficient representation of long and short term dependencies …
Study on memory effects in RNNs learning temporal data.
problem Understanding memory effects in RNNs for temporal data learning.
method Mathematical analysis of continuous-time linear RNNs, focusing on approximation and optimization dynamics.
result Long-term memory requires a large number of neurons and slows down training.
Nonlinear RNNs' memory capacity varies widely, making it impractical.
problem The usefulness of memory capacity as a metric for linear RNNs is questioned.
method Analysis of random nonlinear RNNs with varying input scales.
result Memory capacity of nonlinear RNNs is arbitrary and impractical.
Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.
problem Vanishing gradients in recurrent networks when capturing long-term dependencies.
method Formal analysis of self-attention's effect on gradient propagation, proposing a relevancy screening mechanism.
result Self-attention mitigates vanishing gradients in recurrent networks, providing guarantees.
This paper introduces a hierarchical associative memory model with multiple layers.
problem Limitations of traditional associative memory models with only one hidden layer.
method Develops a fully recurrent model with arbitrary layers, including locally connected ones, and a corresponding energy function.
result The model can dynamically assemble memories using weights from lower layers and higher layers' rules.
Deep neural networks solve stochastic control problems with delay.
problem Challenges in stochastic control problems with delay due to path-dependence and high dimensions.
method Employing recurrent neural networks (RNNs) to parameterize policies and optimize objectives.
result RNNs, especially LSTMs, efficiently capture path-dependence and outperform feedforward networks in training and performance.
Deep RNNs compute American option prices and deltas efficiently.
problem Computing prices and deltas of high-dimensional American options.
method Two deep RNNs, one for price and one for delta, learn over spacetime.
result Linear time and constant memory cost compared to feedforward networks.
New method reduces memory usage in deep HRNNs by replacing gradient backpropagation with local losses.
problem Memory constraints in training deep hierarchical RNNs.
method Replace gradient backpropagation with locally computable losses in deep HRNNs.
result Memory requirements reduced by a factor exponential in hierarchy depth.
TCNs can approximate complex input-output maps with limited memory.
problem Approximating complex input-output maps with limited memory.
method Proved TCNs can approximate a wide class of input-output maps with arbitrary error tolerance.
result Deep ReLU TCNs can approximate input-output maps with finite memory to arbitrary error.
In this work, we first analyze the memory behavior in three recurrent neural networks (RNN) cells; namely, the simple RNN (SRN), the long short-term memory (LSTM) and the gated recurrent unit (GRU), where the memory is defined as a function that maps previous elements in a sequence to the current output. Our study show…
IAM improves deep RL by selectively storing influential past observations.
problem Training and performance issues with recurrent neural networks in high-dimensional environments.
method Influence-aware memory architecture that restricts input to variables influencing hidden state.
result IAM outperforms standard RNNs in training speed and policy performance.
This paper investigates the role of sparsity in Reservoir Computing networks.
problem Designing efficient Recurrent Neural Networks (RNNs) with hidden recurrent layers.
method Empirical investigation of sparsity in input-reservoir connections and recurrent connections.
result Sparsity, particularly in input-reservoir connections, enhances the network's temporal memory and dimensionality.
New models improve language generation by sharing intermediate states.
problem Language models struggle with past mistakes in sequence generation.
method Integrate second-order terms in hidden-state update, sharing intermediate states.
result Shared parametrization improves language modeling performance.
Tensor networks and RNNs are equivalent, improving wave function encoding.
problem Efficiently encoding quantum states in neural networks.
method Generalized RNN architecture for tensor networks, supporting polynomial time wave function evaluation.
result Tensorial RNNs can encode quantum states with lower bond dimensions and higher accuracy.
HiPPO framework optimizes memory compression for sequential data.
problem Incremental representation of cumulative history in sequential data.
method Optimal polynomial projections for online function approximation.
result HiPPO-LegS achieves state-of-the-art accuracy on MNIST.
Cortical circuits exhibit intricate recurrent architectures that are remarkably similar across different brain areas. Such stereotyped structure suggests the existence of common computational principles. However, such principles have remained largely elusive. Inspired by gated-memory networks, namely long short-term me…
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
Next-gen reservoir computers fail to predict complex processes, highlighting need for better architectures.
problem Predicting complex, non-Markovian processes with recurrent neural networks.
method Lower bound from Fano's inequality and analysis of large probabilistic state machines.
result Next-generation reservoir computers have an error probability at least 60% higher than optimal for highly non-Markovian processes.
NeuroMemFPP uses LSTM to estimate FPP parameters with high accuracy.
problem Estimating parameters of fractional Poisson process with memory and long-range dependence.
method Recurrent Neural Network (RNN), specifically Long Short-Term Memory (LSTM), for parameter estimation.
result The LSTM-based approach reduces MSE by about 55.3% compared to traditional MOM method.
The extension of deep learning towards temporal data processing is gaining an increasing research interest. In this paper we investigate the properties of state dynamics developed in successive levels of deep recurrent neural networks (RNNs) in terms of short-term memory abilities. Our results reveal interesting insigh…
LSTMs and GRUs are the most common recurrent neural network architectures used to solve temporal sequence problems. The two architectures have differing data flows dealing with a common component called the cell state (also referred to as the memory). We attempt to enhance the memory by presenting a modification that w…
Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…
In this paper, we revisit the recurrent back-propagation (RBP) algorithm, discuss the conditions under which it applies as well as how to satisfy them in deep neural networks. We show that RBP can be unstable and propose two variants based on conjugate gradient on the normal equations (CG-RBP) and Neumann series (Neuma…
This research unifies concepts of fading memory in RNNs.
problem Unclear relationships between fading memory concepts in RNNs.
method Unified language and new proofs for fading memory concepts.
result Clarified relationships between fading memory concepts.
Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.
problem Efficiently scaling Transformer architecture to long sequences with limited compute resources.
method Combines shallow Transformer models with recurrent neural networks and persistent memory.
result CRT achieves comparable or superior performance to full-length Transformers with shorter segments and reduced FLOPs.
Vanishing long-term gradients are a major issue in training standard recurrent neural networks (RNNs), which can be alleviated by long short-term memory (LSTM) models with memory cells. However, the extra parameters associated with the memory cells mean an LSTM layer has four times as many parameters as an RNN with the…
Linear recurrent networks explain reinforcement learning performance in partially observable settings.
problem Understanding why linear recurrent networks work in reinforcement learning with partial observability.
method Constructed and studied two linear filters for HMMs and action-controlled HMMs.
result Linear filters serve as sufficient statistics and reduce state ambiguity, explaining empirical reinforcement learning success.
This paper reviews methods to create compact neural networks for IoT applications.
problem Complex deep neural networks are costly and slow, hindering real-world deployment.
method Automatic synthesis of compact, accurate DNN/LSTM models.
result Compact neural networks reduce energy consumption, memory, and inference time.
LEM efficiently models long-term sequences with gradients.
problem Learning long-term sequential dependencies in data.
method Gradient-based multiscale ordinary differential equations system.
result LEM outperforms state-of-the-art models in various tasks.
Recurrent neural networks show state-of-the-art results in many text analysis tasks but often require a lot of memory to store their weights. Recently proposed Sparse Variational Dropout eliminates the majority of the weights in a feed-forward neural network without significant loss of quality. We apply this technique …
It is a well-known fact that adding noise to the input data often improves network performance. While the dropout technique may be a cause of memory loss, when it is applied to recurrent connections, Tikhonov regularization, which can be regarded as the training with additive noise, avoids this issue naturally, though …
Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. Fr…
Paper compares GRU and LSTM for predicting wildfire spread direction.
problem Predicting wildfire spread direction with limited data.
method Comparison of Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) networks.
result GRU performs better for longer time series than LSTM.
Metalearned neural memory improves learning across various tasks.
problem Improving neural network adaptability and memory function.
method Augmenting neural networks with a metalearned external memory mechanism.
result The model achieves strong performance on diverse learning problems.
Memory-Augmented Recurrent Networks improve dialogue coherence by expanding conversation history storage.
problem Fixed-size vectors limit dialogue coherence; attention mechanisms are computationally expensive.
method Introduce Neural Turing Machines (NTMs) to provide flexible and permanent storage for dialogue history.
result Improved perplexity performance compared to existing baselines.