Novel method improves training RNNs by accelerating gradient descent.
problem Vanishing and exploding gradient problems in RNNs training.
method Adaptive stochastic Nesterov accelerated quasi-Newton method.
result Improved performance in training RNNs with low per-iteration cost.
AC-RNN improves RNN for sequence labeling tasks.
problem RNN's exposure bias in maximum-likelihood training.
method Actor-Critic training for RNNs.
result AC-RNN outperforms CRF on NER and CCG tagging.
Transfer learning improves clinical time series prediction with limited data.
problem Training deep RNNs for clinical tasks requires large labeled data and tuning.
method Transfer learning from pre-trained RNNs on multiple tasks to new tasks.
result Features from pre-trained RNNs improve model performance and robustness.
adaQN improves training RNNs with low cost and good performance.
problem Training RNNs is computationally difficult due to vanishing/exploding gradient issues.
method Stochastic quasi-Newton algorithm with L-BFGS updating, low per-iteration cost.
result adaQN is competitive with popular RNN training algorithms on language modeling tasks.
Frequentist method estimates uncertainty in RNNs without altering architecture.
problem Uncertainty quantification in RNNs for decision-making.
method Jackknife resampling and influence functions to estimate variability.
result The method provides theoretical coverage guarantees on uncertainty intervals.
Compact RNNs reduce parameters and improve efficiency.
problem High computational cost of RNNs with large inputs.
method Block-Term Tensor Decomposition (BT-TD) to reduce RNN parameters.
result BT-RNN achieves better accuracy and faster convergence than standard RNNs.
ADMMiRNN solves RNN training issues with stable convergence.
problem Training RNN with stable convergence and avoiding gradient issues.
method Built ADMMiRNN framework on unfolded RNN, providing novel update rules and theoretical analysis.
result ADMMiRNN achieves convergent results and outperforms baselines.
Lyapunov exponents help understand RNN stability.
problem Optimizing RNNs is sensitive to various parameters.
method Use Lyapunov exponents as dynamical system tools.
result Lyapunov spectrum measures training stability.
Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.
Analyzes RNNs using ODEs to map their properties and improve stability.
problem Understanding and improving the stability of RNNs.
method Relates RNNs to ODEs, mapping their properties to integration methods.
result Establishes sufficient conditions for RNN training stability and designs new architectures.
RNNs classify text by accumulating evidence on a low-dimensional manifold.
problem Understanding how RNNs solve text classification tasks.
method Dynamical systems analysis applied to trained RNNs.
result RNNs use a low-dimensional attractor manifold to classify text.
Efficient RNN algorithm guarantees convergence in online learning.
problem Online nonlinear regression with RNNs.
method First-order training algorithm with convergence guarantee.
result The algorithm converges to optimum network parameters.
RNNs struggle with chaotic dynamics due to exploding gradients, but we found a way to optimize training.
problem Challenging training of RNNs with chaotic dynamics due to exploding gradients.
method Relating loss gradients to Lyapunov spectrum to optimize training on chaotic data.
result RNNs with chaotic dynamics always have diverging gradients, while stable ones have bounded gradients.
Transfer learning improves clinical time series prediction with deep RNNs.
problem Training deep neural networks for clinical time series analysis requires large labeled data and expertise.
method Investigated transfer learning scenarios for deep RNNs: domain-adaptation and task-adaptation.
result Pre-trained deep models allow robust, efficient, and data-efficient clinical time series prediction.
R2N2 combines linear VAR and nonlinear RNN models for multivariate time series.
problem Multivariate time series modeling with poor predictive performance or complex models.
method R2N2 (Residual RNN) combines simple linear VAR and complex RNN models.
result R2N2 outperforms VAR and RNN alone, and is faster to train.
Advancements in parallel processing have lead to a surge in multilayer perceptrons' (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using con…
Elman-type RNNs converge to globally optimal solutions in the mean-field regime.
problem Optimizing feature learning in wide RNNs.
method Analysis of gradient descent dynamics and mean-field limits.
result Fixed points of infinite-width dynamics are globally optimal.
Recurrent neural networks (RNNs), particularly long short-term memory (LSTM), have gained much attention in automatic speech recognition (ASR). Although some successful stories have been reported, training RNNs remains highly challenging, especially with limited training data. Recent research found that a well-trained …
Gradual training and gradient clipping improve RNN performance.
problem RNNs are hard to train and prone to overfitting.
method Formulated RNN as a Markov chain, gradually trained, and used layer-wise gradient clipping.
result Improvements in language modeling tasks.
Lyapunov analysis improves RNN performance prediction.
problem Uncertainty in RNN performance prediction due to hyperparameters and architecture.
method Lyapunov spectral analysis of RNNs and Autoencoder-Lyapunov Embedding Learning (AeLLE).
result AeLLE successfully correlates RNN Lyapunov spectrum with accuracy and predicts performance.
PSGD accelerates RNN training, achieving competitive performance.
problem Training recurrent neural networks, especially those with long-term memory requirements.
method Preconditioned stochastic gradient descent (PSGD) algorithm.
result PSGD achieves highly competitive performance on RNN training tasks.
A new RNN architecture reduces model size and improves performance.
problem Overparameterization and resource limitations in RNNs.
method Jointly encodes weight matrices using tensor-train factorization.
result Reduces model size by several orders of magnitude without sacrificing performance.
This paper uses RNN to speed up high-speed channel simulation.
problem Computational intensity of generating eye diagrams in high-speed channels.
method Trains a recurrent neural network (RNN) to generate black-box macromodels.
result Reduces computation time significantly without complex simulations.
Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedde…
Explains RNN and LSTM fundamentals, derives formulas, and addresses training issues.
problem Lack of detailed formulas and unrolling techniques in LSTM and RNN literature.
method Derives canonical RNN and LSTM formulas from differential equations, proposes unrolling technique, addresses training difficulties.
result Provides a comprehensive understanding of RNN and LSTM, including detailed formulas and unrolling techniques.
Reversible RNNs reduce memory usage in training without sacrificing performance.
problem Memory-intensive training of RNNs limits model flexibility.
method Developed a scheme for perfect reversible RNNs with forgetting, reducing memory by 10-15x.
result Achieved comparable performance to traditional models with reduced activation memory cost.
RNNs can store information in parameters and input history, with capacity and trainability being key factors.
problem Understanding and optimizing the capacity and trainability of RNNs.
method Experimental analysis of various RNN architectures, including comparison of training difficulty and per-task parameter capacity.
result RNNs can store an amount of task information linear in the number of parameters (approximately 5 bits per parameter) and can store one real number from input history per hidden unit.
Combines RNN and HMM to improve interpretability of RNNs.
problem Making complex RNNs more understandable and interpretable.
method Integrates RNN with HMM, exploring various combinations.
result RNN and HMM learn complementary information.
SEARNN improves RNN training by incorporating global-local losses.
problem RNNs trained with MLE fail to exploit structured losses and suffer from exposure bias.
method SEARNN introduces global-local losses through test-alike search space exploration.
result SEARNN outperforms MLE on OCR, spelling correction, and machine translation tasks.
RNN-Opt learns neural network optimizers for black-box functions.
problem Optimizing black-box functions with limited information.
method Training RNNs with a discounted regret loss function to optimize real-parameter functions.
result RNN-Opt outperforms existing methods on various black-box optimization tasks.
The paper improves DFA for CNN and RNN training to match BP accuracy.
problem Low accuracy in CNN and RNN training using DFA.
method Divide network into modules, apply DFA within, use sparse backward weight, and incorporate dilated convolution and sparse matrix multiplication.
result Achieves BP-level accuracy in CNN and RNN training.
A new model combines spatial and spectral features for HSI classification.
problem Inefficiency and difficulty in training RNNs for HSI classification.
method Proposes St-SS-pGRU combining shorten RNN, converlusion layer, and parallel-GRU.
result Better performance and robustness in HSI classification.
Optimized parallel RNN training reaches up to 845x speedup.
problem Expensive RNN training through back-propagation through time (BPTT).
method Optimized parallel algorithm \opt based on ELM, leveraging GPU shared memory and QR factorization.
result Up to 845x speedup over sequential training and 20x less time to train.
New method compresses LSTM networks using MPS tensor trains.
problem Challenges in maintaining performance of compressed RNNs.
method Use of MPS tensor trains for LSTM network compression.
result MPS tensor trains outperform MPOs in storage and inference time.
DeepESNs outperform ESN and GRUs in multivariate time-series prediction.
problem Comparing DeepESNs and gated RNNs for multivariate time-series prediction.
method Experimental comparison of DeepESNs and gated RNNs (Gated Recurrent Units, Long Short-Term Memory) on 4 polyphonic music tasks.
result DeepESNs outperform ESN and GRUs in terms of prediction accuracy and efficiency.
Sketch-rnn generates coherent drawings from crude sketches.
problem Creating detailed drawings from simple sketches.
method Recurrent Neural Network (RNN) trained on human-drawn images.
result Generates coherent vector drawings from crude sketches.
Mathematical methods characterize RNNs' asymptotics as hidden units and data grow.
problem Characterize recurrent neural networks' behavior as hidden units and data grow.
method Developed mathematical methods to analyze RNNs' convergence to an infinite-dimensional ODE coupled with a fixed point of a random algebraic equation.
result RNNs converge to an infinite-dimensional ODE coupled with a fixed point of a random algebraic equation.
PBO methods improve RNN performance in learning long-term dependencies.
problem Training RNNs to learn long-term dependencies is challenging.
method Population-based global optimisation (PBO) techniques, including evolution strategies and particle swarm optimisation.
result PBO methods lead to performance improvements in RNNs for volatility forecasting.
SRNNs achieve LSTMs' online regression performance in less time.
problem Training LSTMs is time-consuming.
method First-order training algorithm with linear time complexity.
result SRNNs provide similar regression performance as LSTMs in shorter training time.
Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully u…
RNN(p) improves power consumption forecasts with interpretable models.
problem Improving power consumption forecasts for energy sector decisions.
method RNN(p) models with p time lags, using structured feedbacks.
result RNN(p) models achieve excellent forecasting accuracy and interpretability.
New RNN model forecasts unseen time series with little training data.
problem Lack of data for RNNs to generalize well in time series forecasting.
method Proposes a novel RNN-based model that learns shared feature embeddings over quantised time series.
result Accurately forecasts unseen time series with minimal training data.
Integrates momentum into RNNs to improve training.
problem Vanishing gradient in RNNs during training.
method Established connection between RNN dynamics and GD, integrated momentum.
result MomentumRNNs alleviate vanishing gradient issue and improve convergence.
GPU-optimized ES-RNN boosts time series forecasting speed by 322x.
problem Efficiently forecasting time series data.
method Vectorized GPU implementation of ES-RNN.
result Up to 322x speedup in training time.
Stochastic RNNs classify biological neural network paths with robust error bounds.
problem Classifying biological neural network paths.
method Modelled as a continuous-time stochastic recurrent neural network (RNN) with identity activation function, analysed in the robust regime.
result Generalisation error bound holds with high probability, showing the empirical risk minimiser is the best-in-class hypothesis.
Reverse engineered RNNs reveal line attractor dynamics for sentiment classification.
problem Understanding how recurrent neural networks solve sequential tasks like sentiment classification.
method Dynamical systems analysis to reverse engineer trained RNNs, identifying fixed points and linearized dynamics.
result Trained RNNs converge to low-dimensional line attractor dynamics, providing interpretable solutions.
Interpretable RNN uses sparse recovery for better performance.
problem Interpreting the internal workings of RNNs.
method Sequential Sparse Recovery + SISTA algorithm.
result SISTA-RNN achieves better performance and is more interpretable.
New multiplicative LSTM architecture outperforms RNNs in character prediction.
problem Optimizing and contrasting RNN architectures for better time series modeling.
method Hessian free optimization applied to various RNN architectures including multiplicative LSTM.
result Multiplicative LSTM hybrid architecture outperformed LSTM and multiplicative RNNs.