arXiv research

State spaces of multifactor approximations of nonnegative Volterra processes are linear transformations of the nonnegative orthant.

The method approximates stationary distributions of Markov models by truncating irrelevant states.

Combines pseudo-point and state space approximations for scalable GPs.

Paper tackles reinforcement learning for STL specifications with state history.

Fluid approximations have seen great success in approximating the macro-scale behaviour of Markov systems with a large number of discrete states. However, these methods rely on the continuous-time Markov chain (CTMC) having a particular population structure which suggests a natural continuous state-space endowed with a…

Policy gradient methods with aggregated states can achieve better performance than approximate policy iteration.

A new method learns state and proposal dynamics in state-space models using neural networks.

Develops a regression approach for solving MDPs with general state and action spaces.

We consider the problem of approximate Bayesian parameter inference in non-linear state-space models with intractable likelihoods. Sequential Monte Carlo with approximate Bayesian computations (SMC-ABC) is one approach to approximate the likelihood in this type of models. However, such approximations can be noisy and c…

HiPPO-Prophecy models can learn dynamical systems without fine-tuning.

Efficient RL in large POMDPs with latent determinism and embeddings.

New method for efficient probabilistic deep state-space models.

This paper studies optimal approximation factors in misspecified off-policy RL, identifying key factors under various settings.

Robust state-space radio interferometric imaging using Stochastic Approximation Expectation Maximization

When using reinforcement learning (RL) algorithms to evaluate a policy it is common, given a large state space, to introduce some form of approximation architecture for the value function (VF). The exact form of this architecture can have a significant effect on the accuracy of the VF estimate, however, and determining…

Deep learning enhances active inference for dynamic state spaces.

Real Time Dynamic Programming (RTDP) is an online algorithm based on Dynamic Programming (DP) that acts by 1-step greedy planning. Unlike DP, RTDP does not require access to the entire state space, i.e., it explicitly handles the exploration. This fact makes RTDP particularly appealing when the state space is large and…

Bayesian state and parameter estimation for nonlinear models using variational methods.

Our article considers a Gaussian variational approximation of the posterior density in a high-dimensional state space model. The variational parameters to be optimized are the mean vector and the covariance matrix of the approximation. The number of parameters in the covariance matrix grows as the square of the number …

New algorithms reduce complexity for learning in MDPs with entropy regularization.

We prove a conjecture about approximating Gaussian Processes on one dimension.

HyperX uses reward bonuses to enable efficient exploration in meta-learning.

We seek to learn an effective policy for a Markov Decision Process (MDP) with continuous states via Q-Learning. Given a set of basis functions over state action pairs we search for a corresponding set of linear weights that minimizes the mean Bellman residual. Our algorithm uses a Kalman filter model to estimate those …

Paper develops a new state estimation method for nonlinear systems.

We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy a…

We describe a Markov latent state space (MLSS) model, where the latent state distribution is a decaying mixture over multiple past states. We present a simple sampling algorithm that allows to approximate such high-order MLSS with fixed time and memory costs.

A new method for Gaussian filtering using gradient flows and Wasserstein metrics.

Novel framework proves fast RL convergence in continuous spaces.

Improved state estimation in nonlinear models using amortized backward variational inference.

New RL algorithm tackles large state spaces using optimistic function approximation.

The paper introduces Causal Neural Operators to approximate operators in stochastic analysis.

State2vec improves RL by learning state embeddings that generalize across policies.

New RL method handles large state-action spaces with complex models.

Several approximate policy iteration schemes without value functions, which focus on policy representation using classifiers and address policy learning as a supervised learning problem, have been proposed recently. Finding good policies with such methods requires not only an appropriate classifier, but also reliable e…

Study agnostic RL in large state spaces with weak function approximation.

In this paper we show how to approximate a Heath-Jarrow-Morton dynamics for the forward prices in commodity markets with arbitrage-free models which have a finite dimensional state space. Moreover, we recover a closed form representation of the forward price dynamics in the approximation models and derive the rate of c…

Gaussian state space models have been used for decades as generative models of sequential data. They admit an intuitive probabilistic interpretation, have a simple functional form, and enjoy widespread adoption. We introduce a unified algorithm to efficiently learn a broad class of linear and non-linear state space mod…

State-space models (SSMs) provide a flexible framework for modelling time-series data. Consequently, SSMs are ubiquitously applied in areas such as engineering, econometrics and epidemiology. In this paper we provide a fast approach for approximate Bayesian inference in SSMs using the tools of deep learning and variati…

When using reinforcement learning (RL) algorithms it is common, given a large state space, to introduce some form of approximation architecture for the value function (VF). The exact form of this architecture can have a significant effect on an agent's performance, however, and determining a suitable approximation arch…

Active learning selects inputs for GPSSM to learn latent states.

Maximum likelihood (ML) estimation using Newton's method in nonlinear state space models (SSMs) is a challenging problem due to the analytical intractability of the log-likelihood and its gradient and Hessian. We estimate the gradient and Hessian using Fisher's identity in combination with a smoothing algorithm. We exp…

We present a scalable approach to performing approximate fully Bayesian inference in generic state space models. The proposed method is an alternative to particle MCMC that provides fully Bayesian inference of both the dynamic latent states and the static parameters of the model. We build up on recent advances in compu…

New variational inference approach using Hilbert space for robotic state estimation.

We provide a comprehensive overview and tooling for GP modeling with non-Gaussian likelihoods using state space methods. The state space formulation allows for solving one-dimensional GP models in $\mathcal{O}(n)$ time and memory complexity. While existing literature has focused on the connection between GP regression …

We present a scalable and robust Bayesian inference method for linear state space models. The method is applied to demand forecasting in the context of a large e-commerce platform, paying special attention to intermittent and bursty target statistics. Inference is approximated by the Newton-Raphson algorithm, reduced t…

A new method learns complex dynamical systems from data efficiently.

Empirical mode modeling improves state-space analysis of noisy data.

This work improves policy optimization by maximizing entropy of state distribution, leading to better exploration.