HiPPO-Prophecy models can learn dynamical systems without fine-tuning.
problem Learning dynamical systems in context without fine-tuning parameters.
method Introduced a novel weight construction for SSMs that approximates derivatives of input signals.
result Discrete SSMs can predict the next state of any dynamical system after observing previous states.
This paper reviews deep learning methods for state space models.
problem Analyzing temporal dynamics in dynamical systems.
method Selective review of deep neural network approaches for state space models.
result Unified perspective on discrete and continuous time SSMs.
State space models (SSMs) are a flexible approach to modeling complex time series. However, inference in SSMs is often computationally prohibitive for long time series. Stochastic gradient MCMC (SGMCMC) is a popular method for scalable Bayesian inference for large independent data. Unfortunately when applied to depende…
Study stability of selective SSMs with discontinuous gating.
problem Challenges in stability analysis of selective SSMs with discontinuous gating.
method Passivity and Input-to-State Stability (ISS) analysis of continuous-time selective SSMs.
result Derivation of sufficient conditions for global ISS with respect to the port input.
SSMs combined with neural networks match Transformers in dynamic token selection.
problem Understanding the capabilities of SSMs in dynamic token selection.
method Exploring SSMs combined with fully connected neural networks.
result SSMs combined with nonlinear layers can efficiently solve challenging tasks and estimate functions.
The paper provides a theoretical justification for using stable SSM blocks in deep sequential models.
problem Developing generalization bounds for deep sequential models with varying sequence lengths.
method Using Rademacher contraction and stability constraints, the paper derives a PAC bound that is independent of sequence length.
result The derived PAC bound decreases as the stability of SSM blocks increases, providing theoretical justification for their use.
State representation learning (SRL) in partially observable Markov decision processes has been studied to learn abstract features of data useful for robot control tasks. For SRL, acquiring domain-agnostic states is essential for achieving efficient imitation learning. Without these states, imitation learning is hampere…
KalMamba improves RL efficiency with probabilistic SSMs.
problem Efficiency in learning and inference for probabilistic SSMs in RL.
method Combines Mamba's scalability with Kalman filtering for efficient probabilistic SSMs.
result KalMamba outperforms state-of-the-art SSMs in RL, especially on longer sequences.
Deep SSMs use neural networks to identify complex systems.
problem Identifying nonlinear systems with high uncertainty.
method Deep state space models with neural networks.
result Deep SSMs outperform traditional methods on benchmarks.
State-space models (SSMs) provide a flexible framework for modelling time-series data. Consequently, SSMs are ubiquitously applied in areas such as engineering, econometrics and epidemiology. In this paper we provide a fast approach for approximate Bayesian inference in SSMs using the tools of deep learning and variati…
HOPE improves SSMs for long-memory tasks with robust initialization and training.
problem Improving state-space models for long-memory tasks with robust initialization and training.
method Developed a new parameterization scheme called HOPE using Hankel operators and Markov parameters.
result HOPE improves SSMs' performance on Long-Range Arena tasks and demonstrates non-decaying memory.
Structured State-Space Duality connects SSMs to masked attention.
problem Connecting SSMs and attention mechanisms for efficient modeling.
method Formalizing and generalizing SSD from scalar-identity to diagonal state matrices.
result Diagonal SSMs match training complexity lower bounds and support richer dynamics.
Bayesian approach optimizes in-context learning for state space models.
problem Optimizing in-context learning for state space models.
method Bayesian optimal sequential prediction over latent sequence tasks.
result Bayesian optimal predictor converges to posterior predictive mean.
A new variational method for SSMs improves inference efficiency.
problem Hard variational inference for state space models.
method Proposes variational marginal particle filter (VMPF) based on Rao-Blackwellization.
result VMPF provides tighter variational bounds and sometimes benefits from unbiased reparameterization.
Parallelizes autoregressive generation using VSSM.
problem Autoregressive models' inability to parallelize generation.
method Variational SSM (VSSM) with parallelizable sampling and decoding.
result Parallel generation possible with VSSM.
This work studies learning dynamics in SSMs, linking them to deep linear networks.
problem Lack of theoretical understanding of SSMs, especially in deep state spaces.
method Analyzes learning dynamics of linear SSMs, focusing on frequency domain, and establishes links to deep linear networks.
result Analytical solutions for SSM learning dynamics under mild assumptions, linking to deep linear networks.
dynestyx: A library for probabilistic programming of dynamical systems
problem integrating state-space models into probabilistic programming languages
method a unified interface for specifying priors and performing inference
result principled uncertainty quantification for state and parameters
State space models (SSMs) provide a flexible framework for modeling complex time series via a latent stochastic process. Inference for nonlinear, non-Gaussian SSMs is often tackled with particle methods that do not scale well to long time series. The challenge is two-fold: not only do computations scale linearly with t…
Develops an oblique projection technique to approximate a foliation for non-normal dynamics.
problem Modeling dynamics far from a primary Spectral Submanifold (SSM) in non-normal systems.
method Oblique projection technique based on experimental data.
result Approximates a stable invariant foliation for non-normal dynamics efficiently.
State-space models (SSMs) are a highly expressive model class for learning patterns in time series data and for system identification. Deterministic versions of SSMs (e.g. LSTMs) proved extremely successful in modeling complex time series data. Fully probabilistic SSMs, however, are often found hard to train, even for …
Real-world dynamical systems often consist of multiple stochastic subsystems that interact with each other. Modeling and forecasting the behavior of such dynamics are generally not easy, due to the inherent hardness in understanding the complicated interactions and evolutions of their constituents. This paper introduce…
SSMs can be poisoned with clean labels, leading to generalization failure.
problem The implicit bias of SSMs can be manipulated by including special training examples with clean labels.
method Formal proof and empirical demonstration of the phenomenon.
result SSMs can fail to generalize even with clean labels, due to the inclusion of special training examples.
Improved neural likelihood estimation for SSMs with truncated-SNL.
problem Challenges in parameter inference for state-space models.
method Truncated-SNL: a novel inference algorithm addressing SNL's limitations.
result Truncated-SNL is more accurate, scalable, and sample-efficient.
A new method for analyzing high-dimensional time-series data using deep neural networks.
problem Challenges in modeling high-dimensional time-series data with explicit state and observation processes.
method Deep Direct Discriminative Decoders (D4) for high-dimensional observation processes.
result D4 outperforms traditional SSMs and RNNs in various time-series data applications.
A new method for online VI in SSMs using asymptotic contrast.
problem Lack of functionality for streaming data in standard VI methods for SSMs.
method Propose maximising an IWAE-type variational lower bound on the asymptotic contrast function using stochastic approximation.
result OSIWAE allows for online learning of model parameters and latent states.
SSMs have a built-in bias towards low-frequency components, which can be adjusted.
problem Frequency bias in SSMs affects their performance on long-range sequences.
method Proposed two mechanisms to tune frequency bias: scaling initialization or applying a Sobolev-norm-based filter.
result Tuning frequency bias improves SSMs' performance on long-range sequence learning tasks.
State space models (SSM) have been widely applied for the analysis and visualization of large sequential datasets. Sequential Monte Carlo (SMC) is a very popular particle-based method to sample latent states from intractable posteriors. However, SSM is significantly influenced by the choice of the proposal. Recently Ha…
Maximum likelihood (ML) estimation using Newton's method in nonlinear state space models (SSMs) is a challenging problem due to the analytical intractability of the log-likelihood and its gradient and Hessian. We estimate the gradient and Hessian using Fisher's identity in combination with a smoothing algorithm. We exp…
Paper introduces OMD for ordered state transitions in SSMs.
problem Modeling ordered latent states in dynamic systems.
method Ordered Matrix Dirichlet (OMD) prior over ordered stochastic matrices.
result OMD models recover interpretable ordered latent structure without sacrificing predictive performance.
D-LinOSS models learn to dissipate energy, improving performance on long-range tasks.
problem Representational limitations of LinOSS models in long-range reasoning.
method Introducing Damped Linear Oscillatory State-Space models (D-LinOSS) that learn to dissipate latent state energy on arbitrary time scales.
result D-LinOSS consistently outperforms previous LinOSS methods on long-range learning tasks, achieving faster convergence and reducing hyperparameter search space.
One of the key challenges in identifying nonlinear and possibly non-Gaussian state space models (SSMs) is the intractability of estimating the system state. Sequential Monte Carlo (SMC) methods, such as the particle filter (introduced more than two decades ago), provide numerical solutions to the nonlinear state estima…
This paper analyzes EM algorithm for softmax mixture models in high dimensions.
problem Modeling heterogeneous populations choosing from multiple attributes.
method Comprehensive analysis of the EM algorithm for softmax mixture models (SMMs), proving identifiability and convergence.
result EM algorithm recovers mixture atoms at near-parametric rate under suitable initialization.
Online method for state estimation and parameter learning in SSMs.
problem State estimation and parameter learning in state-space models.
method Stochastic gradient optimization of variational lower bound, using backward decompositions and Bellman recursions.
result Ability to operate online without revisiting historic observations.
Gaussian process state-space models (GP-SSMs) are a very flexible family of models of nonlinear dynamical systems. They comprise a Bayesian nonparametric representation of the dynamics of the system and additional (hyper-)parameters governing the properties of this nonparametric representation. The Bayesian formalism e…
New model learns causal world dynamics from state space models.
problem Lack of causal world models in neural world modeling.
method State Space Models (SSM) with attention mechanisms.
result SSM can learn causal models of environments with equivalent performance.
Online VSMC efficiently learns SSM parameters in streaming data.
problem Parameter learning and latent state inference in state-space models.
method Combines particle methods and variational inference for online learning.
result Online VSMC achieves efficient, entirely on-the-fly parameter estimation and particle proposal adaptation.
Study combines VaR and ES forecasts using MCS to improve risk predictions.
problem Combining VaR and ES forecasts to improve risk predictions under uncertainty.
method Employed Model Confidence Set (MCS) methodology to identify best-performing models and combine their forecasts.
result Proposed combined predictors are robust and pass standard backtests.
Particle Metropolis-Hastings enables Bayesian parameter inference in general nonlinear state space models (SSMs). However, in many implementations a random walk proposal is used and this can result in poor mixing if not tuned correctly using tedious pilot runs. Therefore, we consider a new proposal inspired by quasi-Ne…
New method for efficient probabilistic deep state-space models.
problem Efficient inference for probabilistic deep state-space models.
method Deterministic inference algorithm for ProDSSM with neural network weights.
result Superior balance between predictive performance and computational budget.
New PG samplers improve inference in coupled state-space models.
problem Bayesian inference from multiple time series with shared parameters.
method Marginalized Particle Gibbs samplers for coupled state-space models.
result Improved parameter inference through shared information.
The reconstruction of an object's shape or surface from a set of 3D points plays an important role in medical image analysis, e.g. in anatomy reconstruction from tomographic measurements or in the process of aligning intra-operative navigation and preoperative planning data. In such scenarios, one usually has to deal w…
This paper examines how skip connections prevent rank collapse in sequence models.
problem Rank collapse in sequence models, leading to reduced expressivity and training instabilities.
method Analytical and ablation studies of lambda-skip connections in SSMs.
result A sufficient condition to prevent rank collapse across various architectures.
S2P2 model improves predictive likelihoods for MTPPs.
problem Modeling irregular time intervals in event sequences.
method State-space point process model using deep state-space techniques.
result Empirically, S2P2 achieves state-of-the-art predictive likelihoods.
Study improves ECG analysis accuracy using state space models, self-supervised learning, and patient metadata.
problem Improving quantitative accuracy of ECG analysis using deep learning.
method Explored state space models, self-supervised learning, and patient metadata integration.
result Improved ECG analysis accuracy through these components, no significant advantage from higher sampling rates or longer input sizes.
Structured state space models improve ECG classification and reveal new insights.
problem Improving ECG analysis through deep learning.
method Applying structured state space models to capture long-term dependencies in ECG data.
result SSMs lead to significant improvements in ECG classification over current state-of-the-art.
New BED method handles online inference for partially observed dynamical systems.
problem Optimizing data collection for partially observable, partially online dynamical systems.
method Derived estimators of expected information gain and its gradient for SSMs, using nested particle filters.
result Successfully handles both partial observability and online inference in realistic models.
Improved Mamba model for long-range sequence tasks.
problem Mamba's poor performance on long-range sequential tasks.
method Proposed B2S6, combining block-wise selective dynamics and channel-specific bias.
result Empirically, B2S6 outperforms S4 and S4D on LRA tasks.
Dynamical models estimate and predict the temporal evolution of physical systems. State Space Models (SSMs) in particular represent the system dynamics with many desirable properties, such as being able to model uncertainty in both the model and measurements, and optimal (in the Bayesian sense) recursive formulations e…