Echo state networks with random weights can approximate any continuous system.
problem Approximating continuous dynamical systems using echo state networks.
method Randomly generated internal weights and a sampling procedure for activation functions.
result Echo state networks with random weights can approximate any continuous casual time-invariant operators with high probability.
The paper proposes count echo state networks for forecasting graduate student enrollments.
problem Forecasting graduate student enrollments from historical data.
method Developed hierarchical count echo state networks and compared them to Poisson autoregressions and negative binomial models.
result Hierarchical negative binomial based echo state network is the superior model.
Neuro-inspired recurrent neural network algorithms, such as echo state networks, are computationally lightweight and thereby map well onto untethered devices. The baseline echo state network algorithms are shown to be efficient in solving small-scale spatio-temporal problems. However, they underperform for complex task…
Study analyzes Echo State Network parameters for Rossler attractor dynamics.
problem Understanding the influence of network type on Echo State Network performance.
method Experimental analysis of Echo State Network parameters using Rossler attractor.
result Exploration of how network type affects Echo State Network performance.
Efficient echo state network with explicit memory performs well on benchmark tasks.
problem Training differentiable neural computers is difficult and time-consuming.
method Echo state network with an explicit memory.
result Echo state network can recognize all regular languages, including those contractive networks cannot.
This work introduces a tensor-based method to perform supervised classification on spatiotemporal data processed in an echo state network. Typically when performing supervised classification tasks on data processed in an echo state network, the entire collection of hidden layer node states from the training dataset is …
Ensemble of MFESNs improves macroeconomic forecasting.
problem Improving accuracy in macroeconomic forecasting.
method Hedge and Follow-the-Leader schemes applied to MFESNs.
result Ensemble models outperform individual models in forecasting.
This research unifies concepts of fading memory in RNNs.
problem Unclear relationships between fading memory concepts in RNNs.
method Unified language and new proofs for fading memory concepts.
result Clarified relationships between fading memory concepts.
Many neural networks use the tanh activation function, however when given a probability distribution as input, the problem of computing the output distribution in neural networks with tanh activation has not yet been addressed. One important example is the initialization of the echo state network in reservoir computing…
Echo state networks are powerful recurrent neural networks. However, they are often unstable and shaky, making the process of finding an good ESN for a specific dataset quite hard. Obtaining a superb accuracy by using the Echo State Network is a challenging task. We create, develop and implement a family of predictably…
CESAR improves wind speed and power forecasting for high-resolution simulations.
problem Accurate high-resolution wind forecasting for efficient power grid management.
method A spatio-temporal neural network model using deep convolutional autoencoder and echo state network.
result CESAR provides up to 17% improvement in wind speed and power forecasting compared to best alternatives.
ESN models predict intraday stock returns efficiently.
problem Intraday stock return prediction using machine learning.
method Echo State Network (ESN) models with random parameters.
result ESN models achieve strong forecasting performance efficiently.
Optical ESNs enable flexible, efficient machine learning with reduced energy.
problem Implementing universal computational capabilities in machine learning.
method Optical implementation of ESNs leveraging stimulated Brillouin scattering.
result Efficient, scalable, and memory-capable optical reservoir computing.
Echo state networks are computationally lightweight reservoir models inspired by the random projections observed in cortical circuitry. As interest in reservoir computing has grown, networks have become deeper and more intricate. While these networks are increasingly applied to nontrivial forecasting tasks, there is a …
The study of deep recurrent neural networks (RNNs) and, in particular, of deep Reservoir Computing (RC) is gaining an increasing research attention in the neural networks community. The recently introduced Deep Echo State Network (DeepESN) model opened the way to an extremely efficient approach for designing deep neura…
The memory capacity of linear echo state networks is accurately calculated using new numerical methods.
problem Numerical evaluations of memory capacity in recurrent neural networks often contradict theoretical bounds.
method Developed robust numerical approaches exploiting MC neutrality with respect to the input mask matrix.
result Memory curves fully agree with theory when using the proposed methods.
ESNs with transfer learning predict long-term chaotic patterns in spatiotemporal dynamical systems.
problem Predicting long-term statistical patterns of spatiotemporally chaotic dynamical systems.
method Echo state networks (ESNs) with transfer learning.
result ESNs with transfer learning accurately predict long-term statistical properties of spatiotemporally chaotic PDEs.
ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.
problem Approximating ergodic dynamical systems using ESNs.
method Tikhonov least squares regression on ESNs trained on observations from an ergodic dynamical system.
result ESNs trained with Tikhonov least squares approximate the target function in the L2(μ) norm.
New analysis reveals optimal regularization for ESNs, avoiding double descent.
problem Characterizing and optimizing Echo State Networks (ESNs) for precise bias-variance.
method Random matrix theory applied to ESNs in a teacher-student setting.
result ESNs achieve lower MSE with limited training samples and teacher memory.
Reservoir Computing (RC) is a popular methodology for the efficient design of Recurrent Neural Networks (RNNs). Recently, the advantages of the RC approach have been extended to the context of multi-layered RNNs, with the introduction of the Deep Echo State Network (DeepESN) model. In this paper, we study the quality o…
Reservoir Memory Machines solve benchmark tasks faster than Neural Turing Machines.
problem Training Neural Turing Machines is hard and limits their applicability.
method Proposes Reservoir Memory Machines, combining neural network flexibility with Turing machine capabilities, but with faster training via alignment and linear regression.
result Reservoir Memory Machines solve benchmark tasks as well as Neural Turing Machines but are much faster to train.
Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. Th…
Feedback improves ESN performance by 30-60% across various tasks.
problem Reducing computational complexity in ESNs for complex sequential data processing.
method State feedback to modify the internal reservoir state of ESNs.
result Feedback significantly improves ESN performance, reducing error by 30-60%.
New approach speeds up optimization with repeated gradient steps on same batch.
problem Performance bottlenecks in massive parallel pipelines with large batch sizes.
method Data echoing, taking repeated gradient steps on the same batch.
result Data echoing affords speedups on curvature-dominated part of convergence rate.
Echo state network (ESN) is viewed as a temporal non-orthogonal expansion with pseudo-random parameters. Such expansions naturally give rise to regressors of various relevance to a teacher output. We illustrate that often only a certain amount of the generated echo-regressors effectively explain the variance of the tea…
A new method suppresses echo more effectively with lower latency.
problem Echo cannot be fully removed by linear filters due to nonlinear relationships.
method Uses a modified Conv-TasNet with multiple streams of input signals.
result Efficacy validated in both single-talk and double-talk situations.
Unified treatment of RC in stochastic and deterministic settings.
problem Understanding and generalizing reservoir computing in both deterministic and stochastic contexts.
method Investigation of state-space systems, analysis of fading memory and solution stability, introduction of stochastic echo states.
result Generality of fading memory and solution stability in state-space systems, even without the echo state property.
We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional lo…
New reservoir computing approach handles infinite-dimensional systems.
problem Approximating and generalizing complex input/output systems.
method Randomly generated echo state networks with neural networks.
result Proves universal approximation properties for new class of systems.
Jointly reduces echo, reverberation, and noise using neural networks.
problem Simultaneous reduction of acoustic echo, reverberation, and noise.
method Multichannel Gaussian modeling and neural network for joint optimization of filters.
result Outperforms individual and joint non-spectral models.
New definitions of ESP for quantum reservoir computing handle non-stationary systems.
problem Traditional ESP does not apply to non-stationary systems.
method Introduce two new categories of ESP: non-stationary ESP and subset/subspace ESP.
result Demonstrates correspondence between non-stationary ESP and QRC with NARMA tasks.
State-space systems generate probabilistic dependencies between inputs and outputs.
problem Understanding probabilistic dependencies in state-space systems.
method Introducing a probabilistic framework and proving sufficient conditions for output existence and uniqueness.
result State-space systems can generate probabilistic dependencies, even without functional relations.
There is a large amount of interest in understanding users of social media in order to predict their behavior in this space. Despite this interest, user predictability in social media is not well-understood. To examine this question, we consider a network of fifteen thousand users on Twitter over a seven week period. W…
New technique halves scan time for multi-echo MR images.
problem Slow acquisition of multi-echo magnetic resonance images.
method Structured deep dictionary learning for adaptive reconstruction.
result Scan time reduced by half compared to state-of-the-art.
GANs improve echo frame generation with labeled patches.
problem Automated echocardiography with limited labels.
method Conditional GAN with patch-based discriminator.
result GAN-enhanced echo frames match given segmentation masks.
ESN model helps understand climate event impacts.
problem Understanding complex climate event impacts.
method Feature importance methods for ESNs on spatio-temporal climate data.
result Characterized relationships between Mount Pinatubo eruption variables.
The paper evaluates various machine learning models for predicting industrial aging processes.
problem Accurately predicting industrial aging processes to schedule maintenance efficiently.
method Compared traditional stateless models (linear and kernel ridge regression, feed-forward neural networks) to more complex recurrent neural networks (echo state networks and LSTMs) on synthetic and real-world data.
result Recurrent models produce near perfect predictions when trained on larger datasets and maintain good performance even with domain shifts, while simpler models perform comparably on smaller datasets.
We analyze generalization in deep learning models using random matrix theory.
problem Understanding the generalization error in deep learning models with random feature representations.
method Applying Random Matrix Theory to derive asymptotic generalization error formulas for various architectures.
result Linear ESNs are equivalent to ridge regression with exponentially time-weighted input covariance, revealing an inductive bias towards recent inputs.
Study converts echocardiography views using adversarial models.
problem Limited information sharing among echocardiography views.
method Adversarial generative models trained with LSGAN, PatchGAN, and cGAN.
result Generated views correlate with real views in LV area and length.
Reinforcement learning crypto agent achieves high returns on Bitcoin derivatives.
problem Maximizing returns on volatile cryptocurrency markets.
method Online transfer learning with an echo state network and recurrent reinforcement learning.
result Achieves a total return of 350%, net of transaction costs, over five years.
This work builds an automated anomaly detection method for chaotic time series, and more concretely for turbulent, high-dimensional, ocean simulations. We solve this task by extending the Echo State Network by spatially aware input maps, such as convolutions, gradients, cosine transforms, et cetera, as well as a spatia…
Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cr…
Classifies spherical objects filled with water or air using acoustic echoes and Form Function.
problem Automatic recognition of spherical objects based on acoustic echoes.
method Form Function analysis of wideband sonar data, fed into a Multilayer Perceptron (MLP) classifier.
result Performance of the MLP classifier compared to SVM, highlighting the effectiveness of Form Function.
In this article, we derive a new stepsize adaptation for the normalized least mean square algorithm (NLMS) by describing the task of linear acoustic echo cancellation from a Bayesian network perspective. Similar to the well-known Kalman filter equations, we model the acoustic wave propagation from the loudspeaker to th…
In this paper, the echo state network (ESN) memory capacity, which represents the amount of input data an ESN can store, is analyzed for a new type of deep ESNs. In particular, two deep ESN architectures are studied. First, a parallel deep ESN is proposed in which multiple reservoirs are connected in parallel allowing …
Efficient CV for ESNs improves time series predictions.
problem Lack of CV in time series modeling, especially for ESNs.
method Two-level optimizations for k-fold CV of ESNs. result Proposed CV schemes give better and more stable test performance.
Spatio-temporal data and processes are prevalent across a wide variety of scientific disciplines. These processes are often characterized by nonlinear time dynamics that include interactions across multiple scales of spatial and temporal variability. The data sets associated with many of these processes are increasing …
CogScale benchmarks AI architectures for sequential processing.
problem Evaluating AI architectures' ability to process sequential information efficiently.
method 14 scalable synthetic tasks designed to isolate cognitive and memory abilities at different scales.
result Attention mechanisms and modern state-space models consistently maintain high performance as task difficulty scales.