In this paper, we explore different ways to extend a recurrent neural network (RNN) to a \textit{deep} RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,291 papers · 148 categories
Trend · papers per month
48 results for “deep RNNs”
Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.
problem Proving universality of deep narrow RNNs with bounded widths.
method Analyzing RNNs as dynamical systems, proving universality for deep narrow structures with specific widths.
result Minimum width for universality of deep narrow RNNs is independent of data length.
This paper develops a novel deep recurrent neural network for sequential signal reconstruction.
problem Sequential signal reconstruction from low-dimensional measurements.
method Unfolding a reweighted - minimization algorithm to design a deep recurrent neural network.
result The proposed reweighted-RNN significantly outperforms existing RNN models in sequential frame reconstruction.
Transfer learning improves clinical time series prediction with deep RNNs.
problem Training deep neural networks for clinical time series analysis requires large labeled data and expertise.
method Investigated transfer learning scenarios for deep RNNs: domain-adaptation and task-adaptation.
result Pre-trained deep models allow robust, efficient, and data-efficient clinical time series prediction.
Transfer learning improves clinical time series prediction with limited data.
problem Training deep RNNs for clinical tasks requires large labeled data and tuning.
method Transfer learning from pre-trained RNNs on multiple tasks to new tasks.
result Features from pre-trained RNNs improve model performance and robustness.
Deep RNNs reveal interesting insights about short-term memory dynamics.
problem Understanding the short-term memory capabilities of deep RNNs.
method Investigation of state dynamics in successive levels of deep RNNs.
result Higher layers in a hierarchically organized RNN architecture exhibit longer memory spans.
New kernels boost RNN performance on non-time-series data.
problem Improving performance of RNNs on non-time-series data.
method Extended RNN kernels to complex architectures, developed fast GPU implementation.
result RNN-based classifiers outperform baselines on 90 non-time-series datasets.
Deep learning models predict generalization gaps without specific task or architecture.
problem Predicting when deep learning works across different tasks and architectures.
method Created a dataset of 13,500 neural networks trained on various spiral datasets and parameters. Used this dataset to train predictors for generalization gaps.
result DNNs and RNNs outperform linear models in predicting generalization gaps, with RNNs achieving .
PC-RNN reconstructs MRI images from undersampled data with more details.
problem Recovering fine details from undersampled MRI data.
method Pyramid Convolutional RNN (PC-RNN) with three ConvRNN modules for multi-scale reconstruction.
result PC-RNN outperforms other methods in recovering more details from MRI images.
Improved TDNNs enhance speech recognition with deep kernels and frequency-dependent processing.
problem Shallow TDNN models limit long-term context modeling.
method Deepened kernels with residual connections and spectro-temporal processing.
result Deep kernel TDNNs reduce WER by 6% and further by 9% with frequency-dependent Grid-RNN.
Deep transformer models outperform RNNs in character-level language modeling.
problem Improving character-level language modeling performance.
method A deep (64-layer) transformer model with fixed context and auxiliary losses.
result Achieved state-of-the-art performance on text8 and enwik8 benchmarks.
Deep learning model predicts crop yields using CNN-RNN.
problem Challenges in predicting crop yields due to multiple factors.
method CNN-RNN framework using environmental and management data.
result CNN-RNN model outperformed other methods by 9-8% RMSE.
Deep learning models predict chaotic Lorenz 96 system accurately.
problem Predicting short-term and long-term statistics of a multi-scale chaotic system.
method Reservoir computing (RC-ESN), ANN, RNN-LSTM.
result RC-ESN outperforms ANN and RNN-LSTM for short-term prediction.
Deep RNNs compute American option prices and deltas efficiently.
problem Computing prices and deltas of high-dimensional American options.
method Two deep RNNs, one for price and one for delta, learn over spacetime.
result Linear time and constant memory cost compared to feedforward networks.
This paper stabilizes deep neural networks by efficiently parameterizing RNNs with SVD.
problem Vanishing and exploding gradients in deep neural networks, especially RNNs.
method Parameterize RNN transition matrix using SVD and Householder reflectors for efficiency.
result Spectral-RNN method stabilizes gradients and improves generalization, especially for long-range dependencies.
We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic m…
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
Deep RNN model improves long-term BP prediction accuracy.
problem Long-term accuracy decay in BP prediction models.
method Formulated BP estimation as sequence prediction problem; used multilayered LSTM with bidirectional and residual connections.
result Deep RNN model outperformed traditional models in long-term BP prediction.
Deep Recurrent Electricity Theft Detection in AMI Networks with Random Tuning of Hyper-parameterscs.LG
Deep RNN detects electricity theft in smart grids.
problem Electricity theft in smart grids.
method Generalized deep recurrent neural network (RNN) with gated recurrent unit (GRU) and random hyper-parameter tuning.
result Superior performance compared to existing detectors.
Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization…
Proposes Fusion Recurrent Neural Network for sequence data.
problem Improving sequence learning for practical applications.
method Fusion module and Transport module for sequence data.
result Fusion RNN performs comparably to state-of-the-art RNNs.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
Novel complex RNN improves stability and performance in sequence tasks.
problem Lack of complex representations in deep learning for sequence tasks.
method Developed a complex gated recurrent cell combining complex-valued and norm-preserving state transitions with a gating mechanism.
result Improves stability and convergence properties, performs competitively on various tasks.
Combines CNN and RNN for hierarchical image classification.
problem Hierarchical relations between image categories are not captured by flat classifiers.
method Uses a CNN for feature extraction and an RNN for capturing hierarchical class relations. Incorporates residual learning.
result Hierarchical networks outperform state-of-the-art CNNs on a real-world dataset.
Linear RC shows hierarchical temporal patterns in state signals.
problem Understanding hierarchical temporal representations in deep RNNs.
method Used linear recurrent units and frequency analysis on state signals.
result Linear RC reveals intrinsic hierarchical temporal structure.
GCRN models graph-structured sequences with CNN and RNN.
problem Predicting structured sequences of data.
method Combines CNN and RNN on graphs.
result Improves precision and learning speed.
Improves RNN performance under noisy computations.
problem Power and speed limitations in deep learning with noisy analog circuits.
method Deep Noise Injection training to robustify RNN weights/biases.
result Trained RNNs show more consistent performance under noisy inference.
DeepESNs outperform ESN and GRUs in multivariate time-series prediction.
problem Comparing DeepESNs and gated RNNs for multivariate time-series prediction.
method Experimental comparison of DeepESNs and gated RNNs (Gated Recurrent Units, Long Short-Term Memory) on 4 polyphonic music tasks.
result DeepESNs outperform ESN and GRUs in terms of prediction accuracy and efficiency.
Improved volatility estimation using SV-PF-RNN.
problem Estimating true volatility in the presence of market noise.
method SV-PF-RNN: hybrid neural network and particle filter architecture.
result SV-PF-RNN outperforms basic particle filter.
Introduces -Cell for improved financial volatility forecasting.
problem Improving volatility forecasting in financial markets.
method Combines GARCH and deep learning, incorporating stochastic layers and time-varying parameters.
result Demonstrates superior forecasting accuracy compared to traditional models.
PBO methods improve RNN performance in learning long-term dependencies.
problem Training RNNs to learn long-term dependencies is challenging.
method Population-based global optimisation (PBO) techniques, including evolution strategies and particle swarm optimisation.
result PBO methods lead to performance improvements in RNNs for volatility forecasting.
Study on RNNs' ability to approximate past-dependent Hölder functions and their application to regression.
problem Understanding and optimizing the approximation capacity of RNNs for regression tasks.
method Derivation of upper bounds on RNN approximation error for Hölder smooth functions and application to regression.
result Achievement of minimax optimal prediction error bounds for RNNs under various data assumptions.
Study compares deep learning models for traffic forecasting, highlighting graph elements' impact.
problem Challenges in forecasting spatial-temporal traffic patterns.
method In-depth comparative study of four deep neural network models with different basic elements.
result Graph attention improves long-term predictions in traffic forecasting models.
Deep RNN predicts vehicle license plate auction prices with high accuracy.
problem Predicting the auction price of vehicle license plates with desirable numbers.
method Constructed a deep recurrent neural network (RNN) to predict prices based on license plate characters.
result Deep RNN predictions explain over 80 percent of price variations, significantly outperforming previous models.
Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedde…
Advancements in parallel processing have lead to a surge in multilayer perceptrons' (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using con…
Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully u…
The paper distills a weighted automaton from RNNs for language modeling.
problem Tackles the gap between deep learning and grammatical inference.
method Uses a spectral approach to infer a weighted automaton from a trained RNN.
result Extracted weighted automata are good approximations of the RNNs, validating the approach.
Deep neural networks improve forecasting of non-stationary time-series data.
problem Forecasting non-stationary time-series data with structural breaks and high volatility.
method Evaluation of DNN models including MLP, CNN, LSTM-RNN, and GRU-RNN on 10 Indian financial stocks.
result DNN models show better performance for single-step forecasting but degrade for multi-step forecasting, especially for long forecast periods.
DeepESN models efficiently design deep neural networks for temporal data.
problem Designing efficient deep neural networks for temporal data.
method Hierarchical compositions of recurrent layers.
result Intrinsic properties of state dynamics in deep RNNs.
Integrates momentum into RNNs to improve training.
problem Vanishing gradient in RNNs during training.
method Established connection between RNN dynamics and GD, integrated momentum.
result MomentumRNNs alleviate vanishing gradient issue and improve convergence.
Predict and explain storage failures using RNNs with event extraction.
problem Predict and explain failures in storage environments from time series data.
method Extract anomalous spikes as events, then build an RNN classifier with attention mechanisms.
result Comparable accuracy to traditional RNNs with improved explainability.
Paper proposes a deep learning model to predict stock prices using sentiment analysis.
problem Predicting future stock movement using financial textual and numerical data.
method A blending ensemble deep learning model with two levels of RNNs, LSTM, and GRU followed by a fully connected neural network.
result The model improves prediction accuracy compared to traditional methods.
Deep signature/log-signature FBSDE algorithm improves accuracy and training time.
problem Solving FBSDEs with state and path dependent features.
method Incorporates deep signature/log-signature transformation into RNN model.
result Improves accuracy and training time compared to existing methods.
SIS-RNN improves model flexibility for sequential data.
problem Limited expressive power of existing stochastic RNNs.
method Semi-implicit variational inference for implicit latent representations.
result SIS-RNN outperforms existing methods in various tasks.
Paper reduces dialog policy optimization with RL methods.
problem Sample inefficiency in RL for optimizing dialog policies.
method Two RNNs for prediction and experience replay.
result Reduces dialog episodes by about a third.
Paper tackles OMG-Emotion Challenge with CNN-RNN for dimensional emotion recognition.
problem Dimensional emotion recognition in-the-wild from visual analysis.
method Developed a multi-component CNN-RNN deep neural architecture for AffWild Emotion Database.
result Best architectures for valence and arousal estimation over validation data.
Bayesian RNN model forecasts and quantifies uncertainty in spatio-temporal data.
problem Uncertainty quantification in nonlinear spatio-temporal systems.
method Developed a Bayesian RNN model to forecast and quantify uncertainty rigorously.
result The model maintains forecast accuracy while quantifying uncertainty formally.