Stable recurrent models perform similarly to unstable ones, proving useful for sequence tasks.
problem The lack of stability in recurrent neural networks hinders their practical application.
method Theoretical analysis and empirical testing of stable recurrent models.
result Stable recurrent models can be well approximated by feed-forward networks, performing similarly to unstable ones.
Study predicts colorectal polyp recurrence using medical records and statistical models.
problem Identifying patient characteristics influencing colorectal polyp recurrence.
method Natural language processing for extracting polyp characteristics, Kaplan-Meier curves, Cox proportional hazards modeling, random survival forest models.
result Polyp size, number, location, and patient smoking status significantly influence recurrence risk.
Unified recurrent networks reveal differences in complexity levels of grammars.
problem Understanding the complexity and behavior of recurrent networks.
method Connecting recurrent networks with deterministic finite automata and formal grammars.
result Unified recurrent networks improve performance and match grammars from different complexity levels.
Transformer models outperform recurrent ones in modeling hierarchical data.
problem Modeling hierarchical structure in data.
method Introducing Multiresolution Transformer Networks leveraging self-attention.
result Multiresolution Transformer Networks significantly outperform state-of-the-art models on query suggestion datasets.
Recurrent-DBN models dynamic relational data with interpretable latent structures.
problem Interpreting dynamic relational data with hidden structures.
method Recurrent Dirichlet Belief Network framework with hierarchical latent structures and efficient inference strategy.
result Recurrent-DBN discovers interpretable latent structures and improves link prediction.
Stochastic models fail to outperform standard recurrent networks in sequence modeling.
problem Discrepancy in performance between stochastic and standard recurrent models in sequence modeling.
method Re-examine roles of latent variables, remove restriction on fully factorized output distribution, compare auto-regressive models.
result Standard recurrent models consistently outperform stochastic models in sequence modeling.
Study compares LSTM, Transformer, and Mamba for bladder cancer recurrence analysis.
problem Complex time-dependent data in bladder cancer recurrence analysis.
method Evaluation of LSTM, Transformer, and Mamba models using Cox proportional hazards model.
result LSTM-Cox model outperforms Transformer-Cox and Mamba-Cox models in prediction accuracy.
Improved GRU model with weighted time-delay feedback for long-term dependencies.
problem Modeling long-term dependencies in sequential data.
method Introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism.
result τ-GRU outperforms state-of-the-art models on various tasks.
Proposes a new Deep Recurrent Double Q-Learning model for Atari games.
problem Improving Deep Reinforcement Learning models for Atari games.
method Integrates Double Q-Learning and Recurrent Networks (LSTM, DRQN) for Atari games.
result Demonstrates improved performance in Atari games compared to existing models.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
Batch-normalized RHN improves gradient control in recurrent networks.
problem Gradient vanishing or exploding in recurrent networks.
method Batch normalization applied at each recurrence loop in RHN.
result Batch-normalized RHN converges faster and performs better.
TransformerLSR models longitudinal, recurrent, and survival data jointly.
problem Joint modeling of longitudinal measurements, recurrent events, and survival data with dependencies.
method Transformer-based deep learning framework integrating deep temporal point processes and latent structure representation.
result TransformerLSR effectively models all three components simultaneously, demonstrating necessity and effectiveness through simulations and real-world data.
This work combines recurrent models with diffusion for probabilistic time series forecasting.
problem Scalability and capturing high-dimensional distributions and cross-feature dependencies in time series forecasting.
method Combines recurrent neural networks' efficiency with diffusion models' probabilistic modeling, using stochastic interpolants and conditional generation.
result Offers scalable probabilistic time series forecasting methods.
This work introduces a new model for complex stochastic processes.
problem Difficulties in representing non-stationary distributions with conventional models.
method Recurrent Autoregressive Flows using normalizing flows with recurrent neural connections.
result Demonstrates the effectiveness of the proposed model through experiments.
RecNets use RNNs to process image channels in a compact, recurrent way.
problem Creating efficient neural network architectures for computer vision.
method Introducing RecNets with CRC layers that simulate recurrent processing of image channels.
result RecNets achieve superior size-accuracy trade-off compared to other compact models.
Convex program for estimating nonlinear recurrent models with stability conditions.
problem Estimating parameters in nonlinear recurrent models with stability conditions.
method Formulated a convex program for the estimator of nonlinear recurrent models under stability conditions.
result Sample complexity for the convex program estimator under stable dynamics.
ENRNN uses eigenvalue normalization for short-term memory in RNNs.
problem Vanishing/exploding gradient problem and long-term dependency modeling.
method Eigenvalue normalization of recurrent matrix to simulate short-term memory.
result ENRNN outperforms existing RNN variants in experiments.
Improves sequence modeling with a flow-based recurrent mixture density network.
problem Sequence modeling and sequence-to-sequence mapping applications.
method Generalized recurrent mixture density networks using normalized flow transformations.
result Significantly improved fit to image sequences measured by log-likelihood.
New techniques enforce sparseness in recurrent models, reducing memory usage.
problem Reducing memory usage in recurrent sequence models for NLP.
method Enforcing sparseness upfront in recurrent layers for language modeling and sequence labeling.
result Predefined sparseness leads to similar performance with fewer parameters.
New models improve language generation by sharing intermediate states.
problem Language models struggle with past mistakes in sequence generation.
method Integrate second-order terms in hidden-state update, sharing intermediate states.
result Shared parametrization improves language modeling performance.
Proposes a new RNN for language generation capturing long-range dependencies.
problem Capturing long-range word dependencies and sentence order in text corpora.
method Recurrent Hierarchical Topic-Guided RNN with dynamic deep topic model.
result Outperforms larger-context RNN-based language models and learns interpretable topics.
Proposes a new model for better speech segmentation.
problem Improving speech segmentation accuracy.
method Integrates recurrent explicit duration variables into rSLDS and uses Pólya-gamma augmentation for inference.
result Demonstrates improved segmentation on various datasets.
RotRNN uses rotations to simplify long sequence modelling.
problem Complex initialisation and normalisation schemes in linear recurrent models.
method RotRNN employs rotation matrices to simplify and normalise linear recurrent models.
result RotRNN achieves competitive performance on long sequence modelling datasets.
Recurrent models can produce infinite sequences, causing bias; new methods prevent this.
problem Inconsistency in decoding infinite-length sequences from recurrent language models.
method Defined and proved inconsistency of common decoding algorithms; proposed remedies.
result Proposed methods prevent inconsistency in practice.
Bayesian units improve speech recognition with minimal parameters.
problem Improving speech recognition models with fewer parameters.
method Derived Bayesian recurrent units integrated into deep learning frameworks.
result Adding Bayesian units improves speech recognition performance.
SIS-RNN improves model flexibility for sequential data.
problem Limited expressive power of existing stochastic RNNs.
method Semi-implicit variational inference for implicit latent representations.
result SIS-RNN outperforms existing methods in various tasks.
This work optimizes reservoir computing models by linking recurrence and non-linear dynamics.
problem Understanding how recurrence and non-linear dynamics in cortical networks contribute to their function.
method Transformed time-continuous, recurrent dynamics into an effective feed-forward structure of linear and non-linear temporal kernels.
result Optimal time-series classifiers can be built from random reservoir networks, demonstrating significant performance gains.
We present the multiplicative recurrent neural network as a general model for compositional meaning in language, and evaluate it on the task of fine-grained sentiment analysis. We establish a connection to the previously investigated matrix-space models for compositionality, and show they are special cases of the multi…
Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
Algorithm discovers dynamic cell structures for better neural network performance.
problem Finding optimal neural network architectures for diverse data samples and time steps.
method Combines recurrent and recursive neural networks to dynamically search for customized cell structures.
result Achieves better prediction accuracy compared to existing models.
Study on recurrent neural networks' feature selection and memorization using F1B test.
problem Conflict between feature selection and memorization in sequence learning.
method Flagged-1-Bit (F1B) test, four recurrent network models studied analytically and experimentally.
result Conflict can be resolved by gating mechanism or increasing state dimension.
This paper improves speech synthesis using a DGP with SRU for naturalness.
problem Improving naturalness in synthetic speech.
method Deep Gaussian process with a recurrent architecture using SRU.
result SRU-DGP outperforms other models in naturalness of synthetic speech.
The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while…
Theory explains how recurrent networks remember sequences.
problem Understanding how recurrent networks remember sequences and perform well.
method Mean field theory and random matrix theory applied to RNNs with gating mechanisms.
result Gated RNNs outperform non-gated RNNs in remembering sequences.
The pullback approach to global Finsler geometry is adopted. Three classes of recurrence in Finsler geometry are introduced and investigated: simple recurrence, Ricci recurrence and concircular recurrence. Each of these classes consists of four types of recurrence. The interrelationships between the different types of …
Paper introduces a new model for polyphonic music composition.
problem Creating music with multiple interwoven voices.
method Developed a coupled recurrent model using probabilistic factorization and neural network ideas.
result Trained models for single-voice and multi-voice composition on a large dataset.
Study on biharmonic hypersurfaces with specific recurrent operators in Euclidean space.
problem Characterizing biharmonic hypersurfaces with recurrent operators.
method Analysis of various recurrent operators and their impact on biharmonic hypersurfaces.
result Some well-known recurrent operators play a significant role in making biharmonic hypersurfaces minimal.
Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a re…
Scalable verifier for recurrent neural networks using polyhedral abstractions.
problem Certifying the correctness of recurrent neural networks.
method Combining sampling, optimization, and Fermat's theorem for polyhedral abstractions; gradient descent for refinement.
result Successfully verified challenging recurrent models in various domains.
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domains to answer a certain question. Thus, robust attention mechanisms are essential for powerful VQA mo…
Linear Memory Network separates memory and function in RNNs.
problem Complex transduction problems requiring memory and input-output exploitation.
method Conceptual separation between memory and function, using feedforward and autoencoder components.
result Efficient training and competitive performance on polyphonic music datasets.
Recurrent Neural Networks handle sequential data better than traditional networks.
problem Handling temporal context and gradient flow in training RNNs.
method Explains the structure and challenges of RNNs, including training complexities.
result RNNs improve performance in various sequential tasks.
This paper analyzes the generalization risk of unrolled neural networks using Stein's Unbiased Risk Estimator.
problem Analyzing the generalization risk of unrolled neural networks and its relationship to network design and train sample size.
method Using Stein's Unbiased Risk Estimator (SURE), the paper analyzes the generalization risk with bias and variance components for recurrent unrolled networks, focusing on the degrees-of-freedom (DOF) component and the trace of the end-to-end network Jacobian.
result DOF is well-approximated by the weighted path sparsity of the network under incoherence conditions on the trained weights, and DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes.
The aim of the present paper is to investigate new types of recurrence in Finsler geometry, namely, hyper-generalized recurrence and generalized conharmonic recurrence. The properties of such recurrences and their relations to other Finsler recurrences are studied.
RNN beats Lee-Carter in forecasting mortality rates.
problem Forecasting mortality rates across different demographics.
method Long Short-Term Memory (LSTM) recurrent neural network trained on multiple countries, ages, and sexes.
result RNN model outperforms the Lee-Carter model in mortality rate forecasting.
RANP improves neural processes for sequential data.
problem Capturing temporal order and recurrent structure from sequential data.
method Incorporated ANP into a recurrent neural network.
result RANP outperforms NPs and LSTMs in 1D regression and autonomous-driving tasks.
Transformers can outperform feedforward and recurrent networks due to dynamic sparsity.
problem Understanding when and why Transformers outperform other neural network architectures.
method Analyzing a sequence-to-sequence data generating model with dynamic sparsity, proving sample complexity differences between feedforward, recurrent, and Transformers.
result Transformers can learn dynamic sparsity models with lower sample complexity than feedforward and recurrent networks.
To generalize the notion of recurrent manifold, there are various recurrent like conditions in the literature. In this paper we present a recurrent like structure, namely, \textit{super generalized recurrent manifold}, which generalizes both the hyper generalized recurrent manifold and weakly generalized recurrent mani…