This work optimizes reservoir computing models by linking recurrence and non-linear dynamics.
problem Understanding how recurrence and non-linear dynamics in cortical networks contribute to their function.
method Transformed time-continuous, recurrent dynamics into an effective feed-forward structure of linear and non-linear temporal kernels.
result Optimal time-series classifiers can be built from random reservoir networks, demonstrating significant performance gains.
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation b…
Linear recurrent networks explain reinforcement learning performance in partially observable settings.
problem Understanding why linear recurrent networks work in reinforcement learning with partial observability.
method Constructed and studied two linear filters for HMMs and action-controlled HMMs.
result Linear filters serve as sufficient statistics and reduce state ambiguity, explaining empirical reinforcement learning success.
Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.
problem Understanding memory and forecasting capabilities in networks with dependent inputs.
method Formulated bounds for memory and forecasting capacities in terms of network size and input properties.
result Proved that memory capacity for linear recurrent networks with independent inputs is given by the rank of the controllability matrix.
Improved electricity price forecasting model combining linear and non-linear structures.
problem Day-ahead electricity price forecasting in energy systems.
method Recurrent neural networks with embedded linear structures.
result Approximately 11% higher accuracy than state-of-the-art models.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
The paper examines how NFT valuations correlate with market data and social trends.
problem Predicting NFT valuations based on market data and social trends.
method Utilizes public market data, NFT metadata, and social trends data; employs linear regression and recurrent neural networks.
result Identifies correlations between NFT valuations and various features.
Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics,…
Recurrent neural networks (RNNs) are a widely used tool for modeling sequential data, yet they are often treated as inscrutable black boxes. Given a trained recurrent network, we would like to reverse engineer it--to obtain a quantitative, interpretable description of how it solves a particular task. Even for simple ta…
RotRNN uses rotations to simplify long sequence modelling.
problem Complex initialisation and normalisation schemes in linear recurrent models.
method RotRNN employs rotation matrices to simplify and normalise linear recurrent models.
result RotRNN achieves competitive performance on long sequence modelling datasets.
Scalable verifier for recurrent neural networks using polyhedral abstractions.
problem Certifying the correctness of recurrent neural networks.
method Combining sampling, optimization, and Fermat's theorem for polyhedral abstractions; gradient descent for refinement.
result Successfully verified challenging recurrent models in various domains.
Nonlinear RNNs' memory capacity varies widely, making it impractical.
problem The usefulness of memory capacity as a metric for linear RNNs is questioned.
method Analysis of random nonlinear RNNs with varying input scales.
result Memory capacity of nonlinear RNNs is arbitrary and impractical.
Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
Mathematical study of learning long-term integration in linear RNNs.
problem How do linear recurrent neural networks learn to integrate over long timescales?
method Analytical study of linear RNNs trained to integrate white noise and damped oscillatory filters.
result Learning dynamics are described by low-dimensional effective equations for outlier eigenvalues.
We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Com…
Study on memory effects in RNNs learning temporal data.
problem Understanding memory effects in RNNs for temporal data learning.
method Mathematical analysis of continuous-time linear RNNs, focusing on approximation and optimization dynamics.
result Long-term memory requires a large number of neurons and slows down training.
New approach predicts stock price synchronization using RNNs and LSTMs.
problem Forecasting synchronization of stock prices in the Indian market.
method Utilizing recurrence plots and CRQA for non-linear analysis, RNNs and LSTMs for prediction.
result Accuracy of 0.98 and F1 score of 0.83 in predicting stock price synchronization.
The memory capacity of linear echo state networks is accurately calculated using new numerical methods.
problem Numerical evaluations of memory capacity in recurrent neural networks often contradict theoretical bounds.
method Developed robust numerical approaches exploiting MC neutrality with respect to the input mask matrix.
result Memory curves fully agree with theory when using the proposed methods.
Deep RNNs compute American option prices and deltas efficiently.
problem Computing prices and deltas of high-dimensional American options.
method Two deep RNNs, one for price and one for delta, learn over spacetime.
result Linear time and constant memory cost compared to feedforward networks.
Tensor networks and RNNs are equivalent, improving wave function encoding.
problem Efficiently encoding quantum states in neural networks.
method Generalized RNN architecture for tensor networks, supporting polynomial time wave function evaluation.
result Tensorial RNNs can encode quantum states with lower bond dimensions and higher accuracy.
Deep neural networks solve stochastic control problems with delay.
problem Challenges in stochastic control problems with delay due to path-dependence and high dimensions.
method Employing recurrent neural networks (RNNs) to parameterize policies and optimize objectives.
result RNNs, especially LSTMs, efficiently capture path-dependence and outperform feedforward networks in training and performance.
Linear RNNs exhibit a bias towards shorter memory due to initialization variance.
problem Understanding the performance limitations of RNNs, especially linear ones.
method Kernel regime analysis to show equivalence to 1D-convolutional networks and analyze weightings.
result Linear RNNs with random initialization have a bias towards shorter memory periods.
In this paper, we unravel a fundamental connection between weighted finite automata~(WFAs) and second-order recurrent neural networks~(2-RNNs): in the case of sequences of discrete symbols, WFAs and 2-RNNs with linear activation functions are expressively equivalent. Motivated by this result, we build upon a recent ext…
RNNs are reinterpreted as kernel methods using neural ODEs.
problem Improving generalization and stability of RNNs.
method Connecting RNNs to neural ODEs and reproducing kernel Hilbert spaces.
result RNNs can be viewed as linear functions of a specific feature set.
Develops a tensor network framework to reduce RNN complexity for high-dimensional sequence modeling.
problem Exponential parameter growth in RNNs for large multidimensional data.
method Embeds a multi-linear graph filter in a tensor network architecture to approximate RNN hidden states.
result Demonstrates superior performance and reduced complexity compared to traditional RNNs.
Stability of recurrent models is closely linked with trainability, generalizability and in some applications, safety. Methods that train stable recurrent neural networks, however, do so at a significant cost to expressibility. We propose an implicit model structure that allows for a convex parametrization of stable mod…
New methods improve Reservoir Computing for chaotic time series prediction.
problem Chaotic time series prediction in Reservoir Computing.
method Established Recurrent Kernel limit, introduced Structured Reservoir Computing.
result Structured Reservoir Computing is faster and more memory-efficient.
Recently, studies on deep Reservoir Computing (RC) highlighted the role of layering in deep recurrent neural networks (RNNs). In this paper, the use of linear recurrent units allows us to bring more evidence on the intrinsic hierarchical temporal representation in deep RNNs through frequency analysis applied to the sta…
We analyze generalization in deep learning models using random matrix theory.
problem Understanding the generalization error in deep learning models with random feature representations.
method Applying Random Matrix Theory to derive asymptotic generalization error formulas for various architectures.
result Linear ESNs are equivalent to ridge regression with exponentially time-weighted input covariance, revealing an inductive bias towards recent inputs.
A new approach learns to represent context for nonstationary bandits.
problem Nonstationary contextual bandits where patterns change over time.
method Combines recurrent neural networks with contextual linear bandit algorithm.
result Consistently outperforms handcrafted historical contexts and other methods.
The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past cell state values for a limited window size k. The formulas ar…
Hexagon grid patterns emerge from conformal isometry in grid cell neural networks.
problem Understanding the algebraic, geometric, and topological properties of grid cells.
method Investigating recurrent neural network models of grid cells, focusing on Lie group and Lie algebra representations, conformal isometry, and hexagon periodic patterns.
result Conformal isometry leads to hexagon periodic patterns in grid cell responses and accurate path integration.
The paper develops a convex parameterization for robust RNNs ensuring stability and robustness.
problem Lack of stability and robustness guarantees in RNNs for sequence-to-sequence mapping applications.
method Formulated convex sets of RNNs with stability and robustness guarantees using incremental quadratic constraints.
result The proposed model structure ensures global exponential stability and bounds on incremental ℓ2 gain. Neural networks predict traffic flow in smart cities.
problem Forecasting stochastic and nonlinear traffic flow.
method Various recurrent neural networks trained on intersection data.
result Vector output model with gated recurrent units performed best.
We propose reinforcement learning on simple networks consisting of random connections of spiking neurons (both recurrent and feed-forward) that can learn complex tasks with very little trainable parameters. Such sparse and randomly interconnected recurrent spiking networks exhibit highly non-linear dynamics that transf…
CRUs model irregular time series with continuous hidden states.
problem Handling irregular time intervals in sequential data.
method Continuous Recurrent Units (CRUs) that integrate hidden states via a linear stochastic differential equation.
result CRUs outperform methods based on neural ordinary differential equations in irregular time series interpolation.
Study compares MoE and RNN models for stock price prediction across volatility profiles.
problem Improving stock price prediction accuracy across different volatility levels.
method Dynamic Mixture of Experts model combining RNN and linear models, adjusting weights through a gating network.
result MoE model outperforms individual models in reducing prediction errors.
Traditional convolutional layers extract features from patches of data by applying a non-linearity on an affine function of the input. We propose a model that enhances this feature extraction process for the case of sequential data, by feeding patches of the data into a recurrent neural network and using the outputs or…
SKOLR uses linear RNNs to approximate Koopman operators for time-series forecasting.
problem Nonlinear dynamical system analysis and time-series forecasting with infinite-dimensional Koopman operators.
method Established a connection between Koopman operator approximation and linear RNNs, integrating learnable spectral decomposition and MLP.
result SKOLR delivers exceptional performance in various forecasting benchmarks and dynamical systems.
Proposes a new model for better speech segmentation.
problem Improving speech segmentation accuracy.
method Integrates recurrent explicit duration variables into rSLDS and uses Pólya-gamma augmentation for inference.
result Demonstrates improved segmentation on various datasets.
Central Pattern Generators (CPGs) are biological neural circuits capable of producing coordinated rhythmic outputs in the absence of rhythmic input. As a result, they are responsible for most rhythmic motion in living organisms. This rhythmic control is broadly applicable to fields such as locomotive robotics and medic…
Recent work has shown that topological enhancements to recurrent neural networks (RNNs) can increase their expressiveness and representational capacity. Two popular enhancements are stacked RNNs, which increases the capacity for learning non-linear functions, and bidirectional processing, which exploits acausal informa…
Matrix completion models are among the most common formulations of recommender systems. Recent works have showed a boost of performance of these techniques when introducing the pairwise relationships between users/items in the form of graphs, and imposing smoothness priors on these graphs. However, such techniques do n…
Low dimensional representations of words allow accurate NLP models to be trained on limited annotated data. While most representations ignore words' local context, a natural way to induce context-dependent representations is to perform inference in a probabilistic latent-variable sequence model. Given the recent succes…
Beta Basis Function Neural Network (BBFNN) is a special kind of kernel basis neural networks. It is a feedforward network typified by the use of beta function as a hidden activation function. Beta is a flexible transfer function representing richer forms than the common existing functions. As in every network, the arch…
Vanishing long-term gradients are a major issue in training standard recurrent neural networks (RNNs), which can be alleviated by long short-term memory (LSTM) models with memory cells. However, the extra parameters associated with the memory cells mean an LSTM layer has four times as many parameters as an RNN with the…
How can local-search methods such as stochastic gradient descent (SGD) avoid bad local minima in training multi-layer neural networks? Why can they fit random labels even given non-convex and non-smooth architectures? Most existing theory only covers networks with one hidden layer, so can we go deeper? In this paper, w…