Finite precision RNNs have varying computational power, with LSTMs and ReLU-RNNs being more powerful.
problem Understanding the computational limits of finite precision RNNs for language recognition.
method Comparison of different RNN variants with finite precision and linear computation time.
result LSTMs and ReLU-RNNs are strictly stronger than other RNN variants in terms of computational power.
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
Explains RNN and LSTM fundamentals, derives formulas, and addresses training issues.
problem Lack of detailed formulas and unrolling techniques in LSTM and RNN literature.
method Derives canonical RNN and LSTM formulas from differential equations, proposes unrolling technique, addresses training difficulties.
result Provides a comprehensive understanding of RNN and LSTM, including detailed formulas and unrolling techniques.
A-LSTM improves emotion recognition by better modeling time dependencies.
problem Conventional LSTM's time dependency modeling is limited.
method Proposed A-LSTM for better temporal context modeling in RNNs.
result A-LSTM outperforms conventional LSTM by 5.5% in emotion recognition.
Paper compresses RNNs using HT decomposition for better performance.
problem Large model sizes of RNNs in sequence analysis.
method Hierarchical Tucker (HT) tensor decomposition for model compression.
result HT-LSTM achieves better compression and accuracy than state-of-the-art methods.
LSTMs are explained as a weighted sum of context-independent functions.
problem Vanishing gradients in simple RNNs.
method Decouple LSTM gates from simple RNNs, computing an element-wise weighted sum of context-independent functions.
result Gating mechanism alone performs as well as an LSTM in most settings, suggesting more representational power.
Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully u…
SRNNs achieve LSTMs' online regression performance in less time.
problem Training LSTMs is time-consuming.
method First-order training algorithm with linear time complexity.
result SRNNs provide similar regression performance as LSTMs in shorter training time.
Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization…
We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic m…
Diagonal RNNs improve music modeling performance and speed.
problem Improving symbolic music modeling efficiency and accuracy.
method Introduced diagonal recurrent matrices in RNNs for music modeling.
result Diagonal RNNs achieve better test likelihood and faster convergence.
Paper examines why LSTMs outperform GRUs in language modeling.
problem Why LSTMs outperform GRUs in language modeling and machine translation.
method Used simplified language models (Context-Free and Context-Sensitive) to analyze LSTM and GRU performance.
result LSTMs perform better due to their ability to count using cell states, while GRUs struggle.
We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modeling and phoneme recognition, and outperfor…
Improved traffic forecasting model handles missing data.
problem Short-term traffic forecasting with missing values.
method Proposed SBU-LSTM architecture with bidirectional and unidirectional LSTM.
result Superior performance in accuracy and robustness for network-wide traffic prediction.
Deep learning models predict chaotic Lorenz 96 system accurately.
problem Predicting short-term and long-term statistics of a multi-scale chaotic system.
method Reservoir computing (RC-ESN), ANN, RNN-LSTM.
result RC-ESN outperforms ANN and RNN-LSTM for short-term prediction.
New method compresses LSTM networks using MPS tensor trains.
problem Challenges in maintaining performance of compressed RNNs.
method Use of MPS tensor trains for LSTM network compression.
result MPS tensor trains outperform MPOs in storage and inference time.
New RNN model handles long-term dependencies in irregularly-sampled time series.
problem Handling long-term dependencies in irregularly-sampled time series data.
method Designing ODE-LSTMs that separate memory from continuous-time state.
result ODE-LSTMs outperform other RNN-based models on non-uniformly sampled data with long-term dependencies.
A new LSTM model reduces state updates and improves convergence for long sequences.
problem Vanishing gradient problem in RNNs and slow convergence on long sequences.
method Gaussian-gated LSTM (g-LSTM) with a time gate to control neuron updates.
result The g-LSTM model reduces state updates and computes by at least 10x compared to an equivalent LSTM.
New approach predicts stock price synchronization using RNNs and LSTMs.
problem Forecasting synchronization of stock prices in the Indian market.
method Utilizing recurrence plots and CRQA for non-linear analysis, RNNs and LSTMs for prediction.
result Accuracy of 0.98 and F1 score of 0.83 in predicting stock price synchronization.
High-order RNNs reduce speech recognition errors.
problem Vanishing gradients in RNNs.
method High-order RNNs with multiple connections from previous time steps.
result HORNNs reduce WER by 4.2% and 6.3% over RNNs.
This paper improves human activity recognition using LSTM-RNN models.
problem Improving accuracy in human activity recognition from sensor data.
method Design and training of LSTM-RNN models on WISDM dataset.
result Achieved an accuracy of above 94% and a loss of less than 30% in 500 epochs.
Automatically builds a vehicle passage classifier using LSTM-RNNs.
problem Vehicle passage detection using complex sensor data.
method Automatic construction of a binary classifier based on LSTM-RNNs.
result Demonstrated that automatic RNN training can replace handcrafted classifiers.
Echo optimizes LSTM RNN training on GPUs by reducing memory footprint.
problem Memory bottleneck in LSTM RNN training on GPUs.
method Compiler-based feature map recomputation to estimate and balance memory and execution time.
result Average 1.89X reduction in GPU memory footprint.
Modified LSTM cells iteratively process input, improving model performance.
problem Improving LSTM network performance for RNNs.
method Iterative scheme where LSTM computations are repeated over constant input and cell state values, updating hidden state multiple times.
result Enhanced model performance, 3 times more parameters than original model.
Study compares RNN, LSTM, and BP neural networks for forex rate prediction.
problem Improving real-time forex rate prediction accuracy.
method Analyzes RNN, LSTM, and BP neural networks' characteristics and advantages.
result Provides insights for selecting the best price-prediction model.
Paper explores compression techniques for LSTM networks in language modeling.
problem High space complexity and inference time of LSTM networks in language modeling.
method Pruning, quantization, low-rank factorization, tensor train decomposition.
result Compared techniques for model size and fast inference suitability.
Compact RNNs reduce parameters and improve efficiency.
problem High computational cost of RNNs with large inputs.
method Block-Term Tensor Decomposition (BT-TD) to reduce RNN parameters.
result BT-RNN achieves better accuracy and faster convergence than standard RNNs.
Paper addresses LSTM stability for thermal systems using infinity-norm.
problem Stability of LSTM networks in thermal systems.
method Derived ISS∞ condition for LSTM, developed training strategy. result ISS∞-promoted LSTM outperforms other models in thermal system case study. Proposes Fusion Recurrent Neural Network for sequence data.
problem Improving sequence learning for practical applications.
method Fusion module and Transport module for sequence data.
result Fusion RNN performs comparably to state-of-the-art RNNs.
Structural-Jump-LSTM speeds up reading by skipping and jumping text.
problem Sequential inference in RNNs makes reading time linearly dependent on input length.
method Introduces a novel LSTM model with agents for skipping and jumping text.
result Structural-Jump-LSTM achieves best FLOP reduction and maintains or improves accuracy.
ARIMA-LSTM hybrid model predicts stock price correlation coefficients.
problem Predicting future stock price correlation coefficients for portfolio optimization.
method ARIMA-LSTM hybrid model combining ARIMA for linear tendencies and LSTM for non-linear temporal dependencies.
result ARIMA-LSTM model outperforms other models in predicting stock price correlation coefficients.
As deep neural networks continue to revolutionize various application domains, there is increasing interest in making these powerful models more understandable and interpretable, and narrowing down the causes of good and bad predictions. We focus on recurrent neural networks (RNNs), state of the art models in speech re…
Study on RNNs' generalization bounds and variants.
problem Understanding the theoretical limits of RNNs and their variants.
method PAC-Learning framework, spectral norms, and total number of parameters.
result Significantly tighter generalization bound for vanilla RNNs.
Gating units in GRUs and LSTMs create slow modes and control phase-space complexity.
problem Training challenges in RNNs due to exploding or vanishing gradients.
method Random matrix theory and mean-field theory applied to GRUs and LSTMs.
result Gates in GRUs and LSTMs lead to accumulation of slow modes and control phase-space complexity.
This work improves LSTM and GRU training stability and generalization.
problem Training instabilities in LSTMs and GRUs on long sequences.
method Developed a mean field theory to optimize initialization hyperparameters.
result Eliminates or reduces training instabilities and improves generalization.
Improved patient risk stratification with relaxed parameter sharing in clinical time-series data.
problem Learning time-varying relationships in clinical time-series data with limited training data.
method Proposed a novel RNN formulation based on a mixture model with relaxed parameter sharing over time.
result Relaxed parameter sharing leads to improved patient risk stratification performance in settings with limited data.
Proposes ECLSTM for more accurate RUL estimation from time series data.
problem Predicting Remaining Useful Life (RUL) from multivariate time series data.
method Embedded Convolutional LSTM (ECLSTM) with automated hyperparameter optimization.
result ECLSTM outperforms state-of-the-art approaches on benchmark data sets.
A new CNN approach for time series forecasting outperforms traditional RNNs.
problem Time series forecasting using conventional RNNs.
method Temporally folded convolutional neural networks (TFC's) for sequence forecasting.
result TFC's outperform conventional RNNs on sequential MNIST and JSB chorals datasets.
Binary and ternary weights simplify RNNs for mobile devices.
problem Complexity and memory intensity of RNNs on mobile devices.
method Learn binary and ternary weights during training.
result Significant memory saving and inference speedup on ASIC platform.
Model uses RNNs to forecast similar time series groups.
problem Forecasting similar time series databases with traditional methods.
method Time series clustering and LSTM networks.
result Outperforms baseline LSTM model and other methods in forecasting competitions.
Proposes POPQORN to quantify robustness of RNNs, including LSTMs and GRUs.
problem Vulnerability to adversarial attacks in recurrent neural networks.
method Develops POPQORN, a general algorithm to quantify robustness of RNNs.
result Demonstrates effectiveness on different network architectures and shows insights from robustness quantification on individual steps.
OSTSC improves RNN performance on imbalanced time series data.
problem Improving classification accuracy on imbalanced time series data.
method Over-sampling approach for time series classification in R.
result OSTSC increases AUC from 0.543 to 0.784 on high frequency trading dataset.
Paper uses RNNs for more accurate indoor WiFi localization.
problem Accurate indoor WiFi localization using RSSI measurements.
method Proposes recurrent neural networks (RNNs) for trajectory positioning of RSSI data.
result Achieves an average localization error of 0.75 m with 80% under 1 m, outperforming conventional algorithms.
Two tricks reduce LSTM complexity and speed up training.
problem Training large LSTM networks is computationally expensive.
method Matrix factorization and partitioning of LSTM components.
result Significantly faster training with fewer parameters.
A minimal RNN model learns to generate new characters from a single example.
problem Learning efficiency and compositional ability in AI.
method Minimal RNN model with one-shot inference mechanism.
result The model can recombine previously learned substructures to generate new characters.
Time-warping improves RNN transfer learning for diverse time scales.
problem Transfer learning for RNNs with varying time scales.
method Time-warping rescales time in LSTM models for better transfer.
result Time-warping maintains accuracy in transferring RNNs between different time scales.
Study compares LSTM and Transformer models in financial time series prediction.
problem Comparing LSTM and Transformer models for financial time series prediction.
method Various LSTM-based and Transformer-based models compared on financial tasks; DLSTM and new Transformer architecture designed.
result Transformer-based models show limited advantage in absolute price sequence prediction, while LSTM-based models perform better on difference sequences.
Bayesian model predicts sequences better than LSTMs by identifying underlying rules.
problem Current RNNs struggle to generalize from limited training data and identify underlying rules in sequences.
method Bayesian model that learns underlying concepts from sequences and generalizes to new data.
result Bayesian model predicts sequences better than traditional LSTMs.