New deep ESN architectures improve memory capacity and prediction accuracy.
problem Improving memory capacity and prediction accuracy of ESNs.
method Two new deep ESN architectures: parallel and series. Analysis of memory capacity and prediction accuracy.
result Parallel deep ESNs have equivalent memory capacity to shallow ESNs, while series deep ESNs have smaller memory capacity.
We propose a physics-informed Echo State Network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional lo…
ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.
problem Approximating ergodic dynamical systems using ESNs.
method Tikhonov least squares regression on ESNs trained on observations from an ergodic dynamical system.
result ESNs trained with Tikhonov least squares approximate the target function in the L2(μ) norm.
Echo State Networks (ESNs) are recurrent neural networks that only train their output layer, thereby precluding the need to backpropagate gradients through time, which leads to significant computational gains. Nevertheless, a common issue in ESNs is determining its hyperparameters, which are crucial in instantiating a …
New analysis reveals optimal regularization for ESNs, avoiding double descent.
problem Characterizing and optimizing Echo State Networks (ESNs) for precise bias-variance.
method Random matrix theory applied to ESNs in a teacher-student setting.
result ESNs achieve lower MSE with limited training samples and teacher memory.
Feedback improves ESN performance by 30-60% across various tasks.
problem Reducing computational complexity in ESNs for complex sequential data processing.
method State feedback to modify the internal reservoir state of ESNs.
result Feedback significantly improves ESN performance, reducing error by 30-60%.
Echo State Networks (ESN) are a class of Recurrent Neural Networks (RNN) that has gained substantial popularity due to their effectiveness, ease of use and potential for compact hardware implementation. An ESN contains the three network layers input, reservoir and readout where the reservoir is the truly recurrent netw…
Efficient cross-validation method for Echo State Networks reduces validation time complexity.
problem Hyper-parameter tuning and validation for Echo State Networks (ESNs).
method Proposes several k-fold cross-validation schemes for ESNs with an efficient algorithm. result Cross-validation of ESNs can be done for the same time complexity as a single split validation.
Optical ESNs enable flexible, efficient machine learning with reduced energy.
problem Implementing universal computational capabilities in machine learning.
method Optical implementation of ESNs leveraging stimulated Brillouin scattering.
result Efficient, scalable, and memory-capable optical reservoir computing.
Echo state networks are powerful recurrent neural networks. However, they are often unstable and shaky, making the process of finding an good ESN for a specific dataset quite hard. Obtaining a superb accuracy by using the Echo State Network is a challenging task. We create, develop and implement a family of predictably…
ESNs with transfer learning predict long-term chaotic patterns in spatiotemporal dynamical systems.
problem Predicting long-term statistical patterns of spatiotemporally chaotic dynamical systems.
method Echo state networks (ESNs) with transfer learning.
result ESNs with transfer learning accurately predict long-term statistical properties of spatiotemporally chaotic PDEs.
New method combines long-memory reservoirs for accurate dengue forecasting from short data.
problem Accurate dengue forecasting from short, noisy, non-stationary, and nonlinear data.
method Fractional ESN and Wavelet ESN frameworks integrating long-term memory.
result fESN and wESN outperform baselines in multiple dengue datasets and forecasting horizons.
ESN model helps understand climate event impacts.
problem Understanding complex climate event impacts.
method Feature importance methods for ESNs on spatio-temporal climate data.
result Characterized relationships between Mount Pinatubo eruption variables.
Machine learning predicts dam-break flood wave behavior accurately.
problem Predicting long-term wave behavior in dam-break floods.
method Solved Saint-Venant equations using Lax-Wendroff scheme, trained RC-ESN with flow depth data.
result RC-ESN model predicts 286 time-steps ahead with RMSE < 0.01, outperforming LSTM.
Efficient CV for ESNs improves time series predictions.
problem Lack of CV in time series modeling, especially for ESNs.
method Two-level optimizations for k-fold CV of ESNs. result Proposed CV schemes give better and more stable test performance.
Echo state network (ESN) is viewed as a temporal non-orthogonal expansion with pseudo-random parameters. Such expansions naturally give rise to regressors of various relevance to a teacher output. We illustrate that often only a certain amount of the generated echo-regressors effectively explain the variance of the tea…
We analyze generalization in deep learning models using random matrix theory.
problem Understanding the generalization error in deep learning models with random feature representations.
method Applying Random Matrix Theory to derive asymptotic generalization error formulas for various architectures.
result Linear ESNs are equivalent to ridge regression with exponentially time-weighted input covariance, revealing an inductive bias towards recent inputs.
Recurrent neural networks (RNNs) have drawn interest from machine learning researchers because of their effectiveness at preserving past inputs for time-varying data processing tasks. To understand the success and limitations of RNNs, it is critical that we advance our analysis of their fundamental memory properties. W…
Long-lead forecasting for spatio-temporal systems can often entail complex nonlinear dynamics that are difficult to specify it a priori. Current statistical methodologies for modeling these processes are often highly parameterized and thus, challenging to implement from a computational perspective. One potential parsim…
ESN models predict intraday stock returns efficiently.
problem Intraday stock return prediction using machine learning.
method Echo State Network (ESN) models with random parameters.
result ESN models achieve strong forecasting performance efficiently.
Deep models predict intraday electricity prices accurately.
problem Accurately forecasting intraday electricity prices.
method Two deep time series probabilistic models using ESNs with stochastic disturbances and copulas.
result Deep distributional models provide accurate short-term probabilistic price forecasts.
Deep learning models predict chaotic Lorenz 96 system accurately.
problem Predicting short-term and long-term statistics of a multi-scale chaotic system.
method Reservoir computing (RC-ESN), ANN, RNN-LSTM.
result RC-ESN outperforms ANN and RNN-LSTM for short-term prediction.
HFNO enhances interpretability of turbulent flows through parallel wavenumber bin processing.
problem Opaque inner workings of Fourier Neural Operators (FNOs) hinder physical interpretability.
method Introduces HFNO, a novel FNO-based architecture that processes wavenumber bins in parallel, enhancing interpretability.
result HFNO decomposes turbulent flows across various scales, enabling increased interpretability and multiscale modeling.
Spatially aware ESN detects anomalies in chaotic time series.
problem Automated anomaly detection in chaotic time series, especially turbulent ocean simulations.
method Extended Echo State Network with spatially aware input maps and loss function.
result Spatial ESN reduces anomaly detection to thresholding of prediction error.
SORSCNs improve nonstationary data modeling by self-organizing and adjusting network parameters.
problem Nonstationary data challenges traditional models in continuous learning.
method SORSCNs autonomously adjust network parameters and structure in real-time using adaptive algorithms.
result SORSCNs outperform other models in generalizing to nonstationary data.
New framework analyzes temporal features in state space models.
problem Understanding temporal dependencies in data streams.
method Proposes a framework for rigorous analysis of state representations in ESNs, using temporal feature spaces and kernel machines.
result Phase transition in kernel richness for cycle reservoir topology.
Study shows structured reservoirs improve deep ESN performance.
problem Improving performance of deep reservoir computing networks.
method Investigated structured reservoir topologies in deep ESNs.
result Structured reservoirs significantly enhance predictive performance.
We propose an experimental comparison between Deep Echo State Networks (DeepESNs) and gated Recurrent Neural Networks (RNNs) on multivariate time-series prediction tasks. In particular, we compare reservoir and fully-trained RNNs able to represent signals featured by multiple time-scales dynamics. The analysis is perfo…
Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence lear…
Spatio-temporal data and processes are prevalent across a wide variety of scientific disciplines. These processes are often characterized by nonlinear time dynamics that include interactions across multiple scales of spatial and temporal variability. The data sets associated with many of these processes are increasing …
We introduce topological parallelisms of oriented lines (briefly called oriented parallelisms). Every topological parallelism (of lines) on PG(3,R) gives rise to a parallelism of oriented lines, but we show that even the most homogeneous parallelisms of oriented lines other than the Clifford parallelism do not necessar…
Parallelizes MCTS for continuous domains using leaf and root parallelization.
problem Solving challenging tasks in continuous domains using MCTS.
method Extends existing parallelization strategies to continuous domains, focusing on leaf and root parallelization.
result Proposes two final selection strategies for continuous states in root parallelization.
The paper parallelizes HMM inference for efficient long-term computations.
problem Efficiently computing inference in long-term hidden Markov models.
method Parallelization using associative elements and operators for sum-product and max-product algorithms.
result The proposed parallel algorithms are computationally efficient for long time horizons.
New rational parallelisms found on complex manifolds that are not flat.
problem Finding non-flat rational parallelisms on complex manifolds.
method Examined rational parallelisms on compact complex manifolds, discovering non-flat examples.
result Discovered rational parallelisms on compact complex manifolds that are not flat.
Introduces a natural parallel translation for navigation data.
problem Navigation data geometric representation and parallelism.
method Introduces a natural parallel translation using Riemannian parallelism.
result The natural parallel translation preserves the Randers norm and has a finite-dimensional holonomy group.
We prove a conjecture formulated by Pablo M. Chacon and Guillermo A. Lobos in [Pseudo-parallel Lagrangian submanifolds in complex space forms, Differential Geom. Appl.] stating that every Lagrangian pseudo-parallel submanifold of a complex space form of dimension at least 3 is semi-parallel.
This study compares parallel SMC and MCMC for Bayesian deep learning, showing SMC parallel is faster.
problem Efficiently performing Bayesian deep learning with parallel computing.
method Compared sequential Monte Carlo (SMC) and Markov chain Monte Carlo (MCMC) in parallel settings.
result Parallel SMC achieves similar convergence as a single SMC but with reduced communication time.
The paper explores parallel 1-forms on special Finsler manifolds and their properties.
problem Investigating parallel 1-forms on specific Finsler manifolds.
method Analyzing Landsberg manifolds, metrizability freedom, and specific Finsler metrics.
result Landsberg surfaces with parallel 1-forms are necessarily Berwaldian, and the metrizability freedom is at least 2.
This paper surveys parallel submanifolds in Riemannian and pseudo-Riemannian manifolds.
problem Understanding parallel submanifolds in Riemannian and pseudo-Riemannian manifolds.
method Comprehensive survey of parallel submanifolds.
result Extrinsic invariants of parallel submanifolds do not vary from point to point.
We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P process…
Characterizes regular parallelisms in 3D space with 2-torus action.
problem Characterizing regular parallelisms in 3D space with 2-torus action.
method Characterization using compactness, equivalence relations, and properties of complex vector spaces.
result There is a 1-dimensional subtorus fixing every parallel class, leading to 2- or 3-dimensional regular parallelisms.
Paper studies second order symmetric parallel tensors in generalized f.pk-space forms.
problem Exploring properties of second order symmetric parallel tensors in generalized f.pk-space forms.
method Analyzes the properties of second order symmetric parallel tensors and deduces the existence or non-existence of certain tensors and hypersurfaces.
result There does not exist second order skew-symmetric parallel tensor in f.pk-space form. There is no parallel hypersurface in a generalized f.pk-space form but there is semi-parallel hypersurface.
Classifies simply-connected pluriclosed manifolds with parallel Bismut torsion.
problem Classifying specific types of manifolds with parallel Bismut torsion.
method Complete classification through mathematical analysis.
result Established a splitting theorem for certain manifolds.
Study on generalized ξ-parallel maps in Riemannian geometry.
problem Characterizing and understanding generalized ξ-parallel maps.
method Defined energy functional, derived first variation formula, and Euler-Lagrange equation.
result Established fundamental properties and relationships with harmonic and biharmonic maps.
This paper improves parallel belief propagation for scalable machine learning.
problem Efficient parallelization of belief propagation for large-scale machine learning tasks.
method Use of scalable relaxed schedulers to parallelize belief propagation.
result Our approach outperforms previous methods in scalability and convergence time.
Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
The paper examines parallel one forms on Riemannian and Finslerian manifolds.
problem Existence of parallel one forms on Riemannian and Finslerian manifolds.
method Using Finslerian settings, the paper investigates the existence of parallel one forms on Riemannian manifolds and Finslerian manifolds, proving conditions for their existence and non-existence.
result Conditions for the existence and non-existence of parallel one forms on Riemannian and Finslerian manifolds.
Betten and Riesinger constructed Parallelisms of PG(3,R) with automorphism group SO(3,R) by applying the reducible SO(3,R)-action to a rotational Betten spread. This was generalized by the present author so as to include oriented parallelisms (i.e., p…