Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a re…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “stochastic recurrence”
This work introduces a new model for complex stochastic processes.
problem Difficulties in representing non-stationary distributions with conventional models.
method Recurrent Autoregressive Flows using normalizing flows with recurrent neural connections.
result Demonstrates the effectiveness of the proposed model through experiments.
With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence. However, opposite results are also observed in other domains, where standard recurrent networks often outperform stochastic models. To better understand this discrepancy, we re-examine the role…
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The c…
New algorithm guarantees optimal convergence rate for stochastic optimization.
problem Optimal convergence rate for stochastic optimization algorithms.
method Regularized versions of Minimization by Incremental Surrogate Optimization (MISO) with arbitrary recurrent data sampling.
result Expected optimality gap converges at under general recurrent sampling schemes.
Researchers study heavy-tail properties of SGD using stochastic recurrence equations.
problem Analyzing heavy-tail properties of Stochastic Gradient Descent (SGD).
method Modeling SGD iterations as multivariate affine stochastic recursions and applying the theory of irreducible-proximal (i-p) matrices.
result Extended results of Gürbüzbalaban et al. (2020) by using the theory of i-p matrices.
This work combines recurrent models with diffusion for probabilistic time series forecasting.
problem Scalability and capturing high-dimensional distributions and cross-feature dependencies in time series forecasting.
method Combines recurrent neural networks' efficiency with diffusion models' probabilistic modeling, using stochastic interpolants and conditional generation.
result Offers scalable probabilistic time series forecasting methods.
Deep neural networks solve stochastic control problems with delay.
problem Challenges in stochastic control problems with delay due to path-dependence and high dimensions.
method Employing recurrent neural networks (RNNs) to parameterize policies and optimize objectives.
result RNNs, especially LSTMs, efficiently capture path-dependence and outperform feedforward networks in training and performance.
Stochastic recurrent neural networks with latent random variables of complex dependency structures have shown to be more successful in modeling sequential data than deterministic deep models. However, the majority of existing methods have limited expressive power due to the Gaussian assumption of latent variables. In t…
DeepBayes uses neural networks to efficiently estimate parameters in complex dynamical models.
problem Estimating parameters in stochastic, nonlinear dynamical models is challenging.
method DeepBayes leverages deep recurrent neural networks to learn an estimator that minimizes mean-squared error.
result DeepBayes achieves asymptotically equivalent performance to Bayesian estimation methods.
In this paper, we show that the recent integration of statistical models with deep recurrent neural networks provides a new way of formulating volatility (the degree of variation of time series) models that have been widely used in time series analysis and prediction in finance. The model comprises a pair of complement…
Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We unify successful ideas from recently pr…
SORSCNs improve nonstationary data modeling by self-organizing and adjusting network parameters.
problem Nonstationary data challenges traditional models in continuous learning.
method SORSCNs autonomously adjust network parameters and structure in real-time using adaptive algorithms.
result SORSCNs outperform other models in generalizing to nonstationary data.
CRUs model irregular time series with continuous hidden states.
problem Handling irregular time intervals in sequential data.
method Continuous Recurrent Units (CRUs) that integrate hidden states via a linear stochastic differential equation.
result CRUs outperform methods based on neural ordinary differential equations in irregular time series interpolation.
The Stochastic Volatility (SV) model and its variants are widely used in the financial sector while recurrent neural network (RNN) models are successfully used in many large-scale industrial applications of Deep Learning. Our article combines these two methods in a non-trivial way and proposes a model, which we call th…
Autonomous robots need to interact with unknown, unstructured and changing environments, constantly facing novel challenges. Therefore, continuous online adaptation for lifelong-learning and the need of sample-efficient mechanisms to adapt to changes in the environment, the constraints, the tasks, or the robot itself a…
This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient desce…
Noise in RNNs promotes flatter minima and more stable dynamics.
problem Understanding and optimizing the training of RNNs with noise.
method Formalizing RNNs as stochastic differential equations and analyzing the effect of noise in the hidden states.
result Noise injection in RNNs leads to flatter minima, more stable dynamics, and improved robustness.
Recurrent iterated function systems (RIFSs) are improvements of iterated function systems (IFSs) using elements of the theory of Marcovian stochastic processes which can produce more natural looking images. We construct new RIFSs consisting substantially of a vertical contraction factor function and nonlinear transform…
Nonlinear RNNs' memory capacity varies widely, making it impractical.
problem The usefulness of memory capacity as a metric for linear RNNs is questioned.
method Analysis of random nonlinear RNNs with varying input scales.
result Memory capacity of nonlinear RNNs is arbitrary and impractical.
Stochastic RNNs classify biological neural network paths with robust error bounds.
problem Classifying biological neural network paths.
method Modelled as a continuous-time stochastic recurrent neural network (RNN) with identity activation function, analysed in the robust regime.
result Generalisation error bound holds with high probability, showing the empirical risk minimiser is the best-in-class hypothesis.
Optimal stock price prediction model using recurrent neural networks with RMSprop optimizer.
problem Stock price prediction using neural networks.
method Comparison of fully connected, convolutional, and recurrent architectures; inclusion of three optimization techniques.
result Single layer recurrent neural network with RMSprop optimizer produces optimal results with validation and test MAE of 0.0150 and 0.0148 respectively.
Enhances RSCNs with hybrid regularization for nonlinear dynamics.
problem Modeling nonlinear dynamic systems with uncertainties.
method Recurrent stochastic configuration networks with hybrid regularization.
result The method outperforms other models in nonlinear system identification and industrial tasks.
Proposes a new RNN for language generation capturing long-range dependencies.
problem Capturing long-range word dependencies and sentence order in text corpora.
method Recurrent Hierarchical Topic-Guided RNN with dynamic deep topic model.
result Outperforms larger-context RNN-based language models and learns interpretable topics.
Study analyzes stock market dynamics using recurrence measures and transitions.
problem Understanding transitions in stock market dynamics during crises.
method Recurrence plots and networks from nonstationary stock market data.
result Recurrence measures capture transitions in stock market dynamics.
Neural networks predict traffic flow in smart cities.
problem Forecasting stochastic and nonlinear traffic flow.
method Various recurrent neural networks trained on intersection data.
result Vector output model with gated recurrent units performed best.
Stochastic optimization's success linked to heavy-tailed noise.
problem Understanding stochastic optimization's success mechanisms.
method Modeling stochastic optimization as random recurrence relations, analyzing multiplicative noise and heavy-tailed behavior.
result Multiplicative noise leads to heavy-tailed stationary behavior in optimization parameters.
DeepRSCN models nonlinear systems using stochastic configurations.
problem Modeling nonlinear dynamic systems efficiently.
method Incrementally constructed deep reservoir computing framework with random parameters and online weight updates.
result DeepRSCN outperforms single-layer networks in efficiency, learning, and generalization.
In this work we compare different batch construction methods for mini-batch training of recurrent neural networks. While popular implementations like TensorFlow and MXNet suggest a bucketing approach to improve the parallelization capabilities of the recurrent training process, we propose a simple ordering strategy tha…
Mack-Net model combines Mack's model with RNNs for better insurance liability estimation.
problem Accurate estimation of insurance liabilities for better financial decision-making.
method Integrates Mack's reserving model with Recurrent Neural Networks (RNNs).
result Improves accuracy of general insurance liability assessment.
PAC-Bayesian bounds for stochastic LTI systems derived.
problem Error bounds for stochastic LTI systems.
method PAC-Bayesian theory applied to autonomous stochastic LTI models.
result Error bounds for stochastic LTI systems derived.
We present an adaptive approach for valuing the European call option on assets with stochastic volatility. The essential feature of the method is a reduction of uncertainty in latent volatility due to a Bayesian learning procedure. Starting from a discrete-time stochastic volatility model, we derive a recurrence equati…
Neural processes (NPs) learn stochastic processes and predict the distribution of target output adaptively conditioned on a context set of observed input-output pairs. Furthermore, Attentive Neural Process (ANP) improved the prediction accuracy of NPs by incorporating attention mechanism among contexts and targets. In …
We propose a recurrent extension of the Ladder networks whose structure is motivated by the inference required in hierarchical latent variable models. We demonstrate that the recurrent Ladder is able to handle a wide variety of complex learning tasks that benefit from iterative inference and temporal modeling. The arch…
Expressivity of Hidden Markov Chains vs. Recurrent Neural Networks from a system theoretic viewpointeess.SY
This paper compares HMC and RNN expressivity using SRT.
problem Comparing expressivity of HMC and RNN models.
method Embed HMC and RNN in a GUM, use SRT to compare structured covariance series.
result Conditions for realizing covariance series by GUM, HMC, or RNN.
We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d…
Proves local convergence of various online and recurrent optimization algorithms.
problem Proves local convergence of online and recurrent optimization algorithms not covered by standard stochastic gradient descent theory.
method Uses a general set of assumptions for learning dynamical systems online, adopting an 'ergodic' viewpoint.
result Local convergence results for online and recurrent optimization algorithms, including RMSProp, NoBackTrack, UORO, Adam, and RTRL.
We present a deep recurrent neural network architecture to solve a class of stochastic optimal control problems described by fully nonlinear Hamilton Jacobi Bellmanpartial differential equations. Such PDEs arise when one considers stochastic dynamics characterized by uncertainties that are additive and control multipli…
We propose a method to infer stochastic low-rank RNNs from neural data.
problem Fitting low-rank RNNs to noisy, stochastic neural data.
method Variational sequential Monte Carlo methods for stochastic low-rank RNNs.
result Lower dimensional latent dynamics compared to state-of-the-art methods.
It has been suggested in 1999 that a certain volume growth condition for geodesically complete Riemannian manifolds might imply that the manifold is stochastically complete. This is motivated by a large class of examples and by a known analogous criterion for recurrence of Brownian motion. We show that the suggested im…
Two new methods improve forecasting of functional time series data.
problem Forecasting of functional time-dependent data.
method Functional Singular Spectrum Analysis (FSFA) based forecasting methods.
result Our methods outperform existing algorithms for periodic stochastic processes.
The study proves theorems about minimal and H-surfaces in 3D space.
problem Characterizing minimal and H-surfaces in 3D space.
method Analytical proofs and stochastic methods.
result Conditions for minimal and H-surfaces in specific spaces.
PAC-Bayesian-Like Error Bound for a Class of Linear Time-Invariant Stochastic State-Space Modelsstat.ML
Paper derives an error bound for stochastic LTI systems.
problem Stochastic LTI systems with inputs in control engineering and econometrics.
method PAC-Bayesian-Like error bound derivation.
result Derived an error bound for stochastic LTI systems.
Recurrent Neural Networks (RNNs) are among the most popular models in sequential data analysis. Yet, in the foundational PAC learning language, what concept class can it learn? Moreover, how can the same recurrent unit simultaneously learn functions from different input tokens to different output tokens, without affect…
This paper develops a spectral theory of Markovian asset pricing models where the underlying economic uncertainty follows a continuous-time Markov process X with a general state space (Borel right process (BRP)) and the stochastic discount factor (SDF) is a positive semimartingale multiplicative functional of X. A key …
Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we prop…
Recursive KalmanNet generalizes well in noisy, out-of-distribution scenarios.
problem Generalization in noisy, out-of-distribution scenarios.
method Recurrent neural network guided by a Kalman filter.
result Recursive KalmanNet performs well in scenarios with different temporal dynamics from training data.
The paper introduces Causal Neural Operators to approximate operators in stochastic analysis.
problem Leveraging temporal structure in non-linear operators for deep learning models.
method Designing a deep learning model framework for infinite-dimensional linear metric spaces.
result Causal Neural Operators can uniformly approximate Hölder or smooth trace class operators.