Efficient RNN algorithm guarantees convergence in online learning.
problem Online nonlinear regression with RNNs.
method First-order training algorithm with convergence guarantee.
result The algorithm converges to optimum network parameters.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “online RNN”
UORO reduces gradient variance in online RNN learning.
problem Improving gradient estimates in online RNN learning.
method Analyzes and proposes variance reduction techniques for UORO.
result Reduces gradient variance both theoretically and practically.
SRNNs achieve LSTMs' online regression performance in less time.
problem Training LSTMs is time-consuming.
method First-order training algorithm with linear time complexity.
result SRNNs provide similar regression performance as LSTMs in shorter training time.
Paper proposes UIS-RNN for fully supervised speaker diarization.
problem Speaker diarization with unknown number of speakers and time-stamped labels.
method Parameter-sharing RNNs with interleaved states, ddCRP for clustering.
result 7.6% diarization error rate on NIST SRE 2000 CALLHOME.
RNN model predicts handwritten characters from accelerometer and gyroscope data.
problem Online handwritten character recognition using sensor data.
method RNN-based neural network trained on gyroscope and accelerometer data.
result High accuracy on test data, achieving character prediction.
Unified framework for efficient online training of RNNs.
problem Efficient and biologically plausible online training of recurrent neural networks.
method Organizes algorithms based on criteria like past vs. future facing, tensor structure, stochastic vs. deterministic, and closed form vs. numerical.
result Algorithms cluster according to criteria, revealing conceptual connections.
Deep learning model predicts online fraud using customer behavior data.
problem Predicting online financial fraud from customer behavior data.
method Recurrent Neural Network (RNN) integrated with Markov Transition Field (MTF).
result The proposed model significantly improves fraud prediction compared to traditional methods.
Convergence Analysis of Real-time Recurrent Learning (RTRL) for a class of Recurrent Neural Networkscs.LG
RTRL optimizes long sequences without truncation, converging to loss minima.
problem Inaccuracies in TBPTT for long sequences.
method Online optimization with exact gradient calculation.
result RTRL converges to loss minima for a class of RNNs.
Proposes a new RNN model for grouped sequential data with varying time intervals.
problem Implicitly models fixed time intervals between observations and lacks group-level effects.
method Mixed membership framework for RNN, learning group-level base parameter.
result Demonstrates dynamic topic modeling with evolving topic distributions over time.
Paper compares RNNs for short-term traffic flow prediction.
problem Accurate short-term traffic flow prediction for smart cities.
method Used three types of RNNs (simple RNN, GRU, LSTM) and compared with other models.
result Simple RNNs and GRU outperform LSTM in accuracy and training time.
New algorithm for online training of Spiking Neural Networks (SNNs).
problem Training Spiking Neural Networks (SNNs) online with BPTT-equivalent gradients.
method Clear separation of spatial and temporal gradient components, derived from biological insights.
result Online training of SNNs with BPTT-equivalent gradients and low time complexity.
Analyzes learning dynamics of RNNs under locality constraints.
problem Understanding learning dynamics in RNNs with locality constraints.
method Dynamical systems theory applied to data-aligned linear RNNs.
result RFLO solutions are restricted to low-rank perturbations of initial parameters.
The paper improves recommendation models by considering user interactions with recommended items.
problem Improving next item prediction in recommendation systems.
method Extending RNN framework with a recommendation action module and state-action fusion module.
result Improved performance on next item prediction compared to baselines.
HiPPO framework optimizes memory compression for sequential data.
problem Incremental representation of cumulative history in sequential data.
method Optimal polynomial projections for online function approximation.
result HiPPO-LegS achieves state-of-the-art accuracy on MNIST.
A novel online GP model captures long-term memory in sequential data.
problem Capturing long-term memory in sequential data online.
method Integrates HiPPO framework into interdomain GP, leveraging time-varying orthogonal projections as inducing variables.
result OHSVGP outperforms existing online GP methods in predictive performance, long-term memory preservation, and computational efficiency.
PV-RNN model uses predictive coding to predict and recognize data sequences.
problem Addressing variational RNNs' challenges in learning meaningful latent representations and transferring future observations.
method Inspired by predictive coding, PV-RNN dynamically changes latent stochasticity and uses prediction errors for information transfer.
result PV-RNN outperforms standard variational Bayes models in predicting unseen sequences and generalizes well.
Paper describes a video surveillance system for highway traffic events.
problem Detecting specific sequences of situations in highway traffic videos.
method Compares RNN and CNN architectures for analyzing video frames and sequences.
result Best architecture performs well in real conditions.
DeepMPC uses neural networks to control complex fluid flows efficiently.
problem Controlling complex fluid flows in real-time is challenging due to high dimensionality and multi-scale dynamics.
method Deep learning, specifically recurrent neural networks (RNNs), embedded in model predictive control (MPC) framework.
result Significant improvements in control performance achieved through online updates to prediction accuracy.
In this paper, we apply neural networks into digital marketing world for the purpose of better targeting the potential customers. To do so, we model the customer online behaviours using dedicated neural network architectures. Starting from user searched keywords in a search engine to the landing page and different foll…
The study predicts how discussions in mental disorder Reddit communities affect users' emotional states.
problem Improving mental health conditions through social support analysis.
method Text embedding techniques and RNNs for predicting emotional tone shifts.
result Users' emotional states can improve due to social support, as evidenced by positive comments following negative posts.
Delayed-RNN approximates stacked and bidirectional RNNs.
problem Improving RNN expressiveness and representational capacity.
method Weight-constrained delayed-RNN, equivalent to stacked-RNNs, with partial acausality.
result Delayed-RNN can approximate stacked and bidirectional RNNs, outperforming them in some tasks.
This study provides benchmarks for different implementations of LSTM units between the deep learning frameworks PyTorch, TensorFlow, Lasagne and Keras. The comparison includes cuDNN LSTMs, fused LSTM variants and less optimized, but more flexible LSTM implementations. The benchmarks reflect two typical scenarios for au…
Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in …
PF-RNNs use particle filtering to model uncertainty in RNNs for better sequential data prediction.
problem Highly variable and noisy sequential data.
method PF-RNNs maintain a latent state distribution as a set of particles, updating with Bayes rule.
result PF-RNNs outperform standard RNNs on various sequence prediction tasks.
In this paper, we explore different ways to extend a recurrent neural network (RNN) to a \textit{deep} RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points …
This paper studies a theoretical pruning method for RNNs to reduce computational costs.
problem High computational costs in recurrent neural networks (RNNs).
method Spectral pruning inspired approach for RNNs.
result Generalization error bounds for compressed RNNs are provided.
Lyapunov analysis improves RNN performance prediction.
problem Uncertainty in RNN performance prediction due to hyperparameters and architecture.
method Lyapunov spectral analysis of RNNs and Autoencoder-Lyapunov Embedding Learning (AeLLE).
result AeLLE successfully correlates RNN Lyapunov spectrum with accuracy and predicts performance.
In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…
Proposes Fusion Recurrent Neural Network for sequence data.
problem Improving sequence learning for practical applications.
method Fusion module and Transport module for sequence data.
result Fusion RNN performs comparably to state-of-the-art RNNs.
Recurrent neural networks (RNNs) are powerful and effective for processing sequential data. However, RNNs are usually considered "black box" models whose internal structure and learned parameters are not interpretable. In this paper, we propose an interpretable RNN based on the sequential iterative soft-thresholding al…
While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time. We consider the case of RNNs with finite precision whose computation time is linear in the input length. Under these limitations, we show that different RNN va…
Paper compresses RNNs for resource-constrained devices.
problem Difficulty deploying RNNs on resource-constrained devices.
method Uses Kronecker product (KP) to compress RNN layers.
result KP compresses RNN layers by 16-38x with minimal accuracy loss.
Understanding and predicting the popularity of online items is an important open problem in social media analysis. Considerable progress has been made recently in data-driven predictions, and in linking popularity to external promotions. However, the existing methods typically focus on a single source of external influ…
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
GPU-optimized ES-RNN boosts time series forecasting speed by 322x.
problem Efficiently forecasting time series data.
method Vectorized GPU implementation of ES-RNN.
result Up to 322x speedup in training time.
Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothnesscs.LG
Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.
Novel method improves training RNNs by accelerating gradient descent.
problem Vanishing and exploding gradient problems in RNNs training.
method Adaptive stochastic Nesterov accelerated quasi-Newton method.
result Improved performance in training RNNs with low per-iteration cost.
Analyzes RNNs using ODEs to map their properties and improve stability.
problem Understanding and improving the stability of RNNs.
method Relates RNNs to ODEs, mapping their properties to integration methods.
result Establishes sufficient conditions for RNN training stability and designs new architectures.
AC-RNN improves RNN for sequence labeling tasks.
problem RNN's exposure bias in maximum-likelihood training.
method Actor-Critic training for RNNs.
result AC-RNN outperforms CRF on NER and CCG tagging.
Elman-type RNNs converge to globally optimal solutions in the mean-field regime.
problem Optimizing feature learning in wide RNNs.
method Analysis of gradient descent dynamics and mean-field limits.
result Fixed points of infinite-width dynamics are globally optimal.
Deep neural networks have shown promising results for various clinical prediction tasks such as diagnosis, mortality prediction, predicting duration of stay in hospital, etc. However, training deep networks -- such as those based on Recurrent Neural Networks (RNNs) -- requires large labeled data, high computational res…
Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization…
New kernels boost RNN performance on non-time-series data.
problem Improving performance of RNNs on non-time-series data.
method Extended RNN kernels to complex architectures, developed fast GPU implementation.
result RNN-based classifiers outperform baselines on 90 non-time-series datasets.
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Toolscs.LG
Lyapunov exponents help understand RNN stability.
problem Optimizing RNNs is sensitive to various parameters.
method Use Lyapunov exponents as dynamical system tools.
result Lyapunov spectrum measures training stability.
GRUwE improves irregular time series prediction with simpler, efficient RNN-based approach.
problem Irregularly sampled multivariate time series prediction challenges.
method Gated Recurrent Unit with Exponential basis functions (GRUwE).
result GRUwE achieves competitive or superior performance compared to recent state-of-the-art methods.
GEN generates millions of valid SMILES with high novelty and property conservation.
problem Generating high-quality, de novo molecules in a known chemical space.
method GEN uses bidirectional RNNs with concatenated sub-models to learn and generate SMILES, with online examination to ensure quality.
result GEN can generate SMILES with 95-98% validity, 85-90% novelty, and 95-99% property conservation.
Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.
problem Proving universality of deep narrow RNNs with bounded widths.
method Analyzing RNNs as dynamical systems, proving universality for deep narrow structures with specific widths.
result Minimum width for universality of deep narrow RNNs is independent of data length.
Hamiltonian RNN controls hidden states gradient for long-term dependencies.
problem Challenges in learning long-term dependencies in RNNs.
method Symplectic discretization of Hamiltonian system to control gradient.
result Hamiltonian RNN outperforms other RNNs without hyperparameter optimization.