Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent net…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “Vanilla RNN”
Study on recurrent neural networks' feature selection and memorization using F1B test.
problem Conflict between feature selection and memorization in sequence learning.
method Flagged-1-Bit (F1B) test, four recurrent network models studied analytically and experimentally.
result Conflict can be resolved by gating mechanism or increasing state dimension.
Study on RNNs' generalization bounds and variants.
problem Understanding the theoretical limits of RNNs and their variants.
method PAC-Learning framework, spectral norms, and total number of parameters.
result Significantly tighter generalization bound for vanilla RNNs.
In this paper, we propose a new Recurrent Neural Network (RNN) architecture. The novelty is simple: We use diagonal recurrent matrices instead of full. This results in better test likelihood and faster convergence compared to regular full RNNs in most of our experiments. We show the benefits of using diagonal recurrent…
Regularizes RNNs to handle long-range dependencies and multiple time scales.
problem Identifying nonlinear dynamical systems with varying time scales and long-range dependencies.
method A simple regularization scheme for vanilla RNNs with ReLU activation.
result Regularized RNNs can solve long-range dependency problems and express slow time scales.
ERNNs evolve hidden states on an ODE's equilibrium manifold to mitigate vanishing and exploding gradients.
problem Vanishing and exploding gradients in RNNs.
method Develop a novel family of RNNs (ERNNs) that evolve hidden states on the equilibrium manifold of an ODE.
result ERNNs achieve state-of-the-art accuracy with 3-10x speedups and 1.5-3x model size reduction.
Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training fo…
In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks und…
Markovian RNN adapts to nonstationary data using HMM for better time series prediction.
problem Nonstationary sequential data in real-life applications.
method Markovian RNN with HMM for regime switching and end-to-end optimization.
result Significant performance gains over vanilla RNN and Markov Switching ARIMA.
Paper uses RNNs for more accurate indoor WiFi localization.
problem Accurate indoor WiFi localization using RSSI measurements.
method Proposes recurrent neural networks (RNNs) for trajectory positioning of RSSI data.
result Achieves an average localization error of 0.75 m with 80% under 1 m, outperforming conventional algorithms.
ParaRNN: An Interpretable and Parallelizable Recurrent Neural Network for Time-Dependent Datastat.ML
ParaRNN improves RNN interpretability and parallelizability for time-dependent data.
problem Limited interpretability and slow training of RNNs.
method Parallelized RNN with additive representation and recurrence features.
result ParaRNN achieves comparable performance to vanilla RNNs but with improved interpretability and efficiency.
Improved stable RNNs trained faster with less expressibility trade-off.
problem Stable recurrent neural networks are hard to train without sacrificing expressibility.
method Implicit model structure with contraction analysis for stable models.
result Significant increase in training speed and model performance.
Paper shows SGD can learn RNNs efficiently for certain functions.
problem Understanding what concept class RNNs can learn and how efficiently.
method Vanilla stochastic gradient descent (SGD) approach.
result RNNs can learn some functions efficiently with polynomial complexity in input length.
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and pe…
ADMMiRNN solves RNN training issues with stable convergence.
problem Training RNN with stable convergence and avoiding gradient issues.
method Built ADMMiRNN framework on unfolded RNN, providing novel update rules and theoretical analysis.
result ADMMiRNN achieves convergent results and outperforms baselines.
This paper uses RNN to speed up high-speed channel simulation.
problem Computational intensity of generating eye diagrams in high-speed channels.
method Trains a recurrent neural network (RNN) to generate black-box macromodels.
result Reduces computation time significantly without complex simulations.
Paper establishes generalization bounds for RNNs and improves existing results.
problem Theoretical understanding and generalization bounds for RNNs.
method New generalization error bound and Rademacher complexity calculation.
result Improved generalization bounds for RNNs, tighter than existing bounds.
Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamicscs.LG
Reverse engineered RNNs reveal line attractor dynamics for sentiment classification.
problem Understanding how recurrent neural networks solve sequential tasks like sentiment classification.
method Dynamical systems analysis to reverse engineer trained RNNs, identifying fixed points and linearized dynamics.
result Trained RNNs converge to low-dimensional line attractor dynamics, providing interpretable solutions.
Proposes POPQORN to quantify robustness of RNNs, including LSTMs and GRUs.
problem Vulnerability to adversarial attacks in recurrent neural networks.
method Develops POPQORN, a general algorithm to quantify robustness of RNNs.
result Demonstrates effectiveness on different network architectures and shows insights from robustness quantification on individual steps.
Much combinatorial optimisation problems constitute a non-polynomial (NP) hard optimisation problem, i.e., they can not be solved in polynomial time. One such problem is finding the shortest route between two nodes on a graph. Meta-heuristic algorithms such as along with mixed-integer programming (MIP) methods …
Structural-Jump-LSTM speeds up reading by skipping and jumping text.
problem Sequential inference in RNNs makes reading time linearly dependent on input length.
method Introduces a novel LSTM model with agents for skipping and jumping text.
result Structural-Jump-LSTM achieves best FLOP reduction and maintains or improves accuracy.
This work improves LSTM and GRU training stability and generalization.
problem Training instabilities in LSTMs and GRUs on long sequences.
method Developed a mean field theory to optimize initialization hyperparameters.
result Eliminates or reduces training instabilities and improves generalization.
We explore the robustness of recurrent neural networks when the computations within the network are noisy. One of the motivations for looking into this problem is to reduce the high power cost of conventional computing of neural network operations through the use of analog neuromorphic circuits. Traditional GPU/CPU-cen…
ARMIN improves memory efficiency and lightness in neural networks.
problem Complex memory addressing and inefficient memory information exploitation in previous MANNs.
method ARMIN uses hidden state for automatic memory addressing and a novel RNN cell for memory integration.
result ARMIN achieves lower computational overhead and similar performances compared to vanilla LSTM.
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-le…
We propose a simple extension to the ReLU-family of activation functions that allows them to shift the mean activation across a layer towards zero. Combined with proper weight initialization, this alleviates the need for normalization layers. We explore the training of deep vanilla recurrent neural networks (RNNs) with…
The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past cell state values for a limited window size . The formulas ar…
Transformers become faster by linearizing self-attention.
problem Quadratic complexity of transformers makes them slow for long sequences.
method Expressed self-attention as a linear dot-product and used matrix product associativity to reduce complexity.
result Linear transformers are up to 4000x faster on long sequences.
We parallelize backpropagation for deep learning models, achieving significant speedups.
problem Sequential dependency in backpropagation limits scalability on parallel systems.
method Reformulated backpropagation as a scan operation, using Blelloch scan algorithm.
result Up to 2.75x speedup on overall training time and 108x on backward pass.
Delayed-RNN approximates stacked and bidirectional RNNs.
problem Improving RNN expressiveness and representational capacity.
method Weight-constrained delayed-RNN, equivalent to stacked-RNNs, with partial acausality.
result Delayed-RNN can approximate stacked and bidirectional RNNs, outperforming them in some tasks.
Paper uses AI for more efficient hedging of financial options.
problem Inefficient hedging in financial models.
method RL agents and GANs for delta hedging.
result RL-based hedging outperforms classic models in Q-world.
Proposes new listwise learning-to-rank models to address rating ties and document relevance.
problem Rating ties and document relevance in existing listwise learning-to-rank models.
method Models ranking as selecting documents from a candidate set based on unique rating levels. Uses a new loss function and adapted RNN model for refining prediction scores.
result Models notably outperform state-of-the-art learning-to-rank models on four public datasets.
Optimal hedging strategies for exotic options using vanilla options.
problem Hedging exotic options with illiquid vanilla options.
method Simple approximations and variational techniques in a market model and stochastic volatility model framework.
result Optimal Delta and Vega hedging strategies can be computed easily.
Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in …
PF-RNNs use particle filtering to model uncertainty in RNNs for better sequential data prediction.
problem Highly variable and noisy sequential data.
method PF-RNNs maintain a latent state distribution as a set of particles, updating with Bayes rule.
result PF-RNNs outperform standard RNNs on various sequence prediction tasks.
In this paper, we explore different ways to extend a recurrent neural network (RNN) to a \textit{deep} RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points …
This paper studies a theoretical pruning method for RNNs to reduce computational costs.
problem High computational costs in recurrent neural networks (RNNs).
method Spectral pruning inspired approach for RNNs.
result Generalization error bounds for compressed RNNs are provided.
Lyapunov analysis improves RNN performance prediction.
problem Uncertainty in RNN performance prediction due to hyperparameters and architecture.
method Lyapunov spectral analysis of RNNs and Autoencoder-Lyapunov Embedding Learning (AeLLE).
result AeLLE successfully correlates RNN Lyapunov spectrum with accuracy and predicts performance.
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
Proposes Fusion Recurrent Neural Network for sequence data.
problem Improving sequence learning for practical applications.
method Fusion module and Transport module for sequence data.
result Fusion RNN performs comparably to state-of-the-art RNNs.
Vanilla GANs are connected to Wasserstein distance for better understanding.
problem Understanding the statistical properties of Vanilla GANs.
method Connecting Vanilla GANs to Wasserstein distance and proving an oracle inequality.
result An oracle inequality for Vanilla GANs in Wasserstein distance is obtained.
Recurrent neural networks (RNNs) are powerful and effective for processing sequential data. However, RNNs are usually considered "black box" models whose internal structure and learned parameters are not interpretable. In this paper, we propose an interpretable RNN based on the sequential iterative soft-thresholding al…
Efficient RNN algorithm guarantees convergence in online learning.
problem Online nonlinear regression with RNNs.
method First-order training algorithm with convergence guarantee.
result The algorithm converges to optimum network parameters.
While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN va…
Paper compresses RNNs for resource-constrained devices.
problem Difficulty deploying RNNs on resource-constrained devices.
method Uses Kronecker product (KP) to compress RNN layers.
result KP compresses RNN layers by 16-38x with minimal accuracy loss.
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
GPU-optimized ES-RNN boosts time series forecasting speed by 322x.
problem Efficiently forecasting time series data.
method Vectorized GPU implementation of ES-RNN.
result Up to 322x speedup in training time.
Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothnesscs.LG
Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.