arXiv research

Algorithm learns mixtures of Markov chains and MDPs from short trajectories.

Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.

Model reduction of Markov processes is a basic problem in modeling state-transition systems. Motivated by the state aggregation approach rooted in control theory, we study the statistical state compression of a discrete-state Markov chain from empirical trajectories. Through the lens of spectral decomposition, we study…

The paper tackles learning optimal predictions from a single trajectory of a stochastic dynamical system.

Reduces identity testing of reversible Markov chains to simpler symmetric chain tests.

We propose a deep generative Markov State Model (DeepGenMSM) learning framework for inference of metastable dynamical systems and prediction of trajectories. After unsupervised training on time series data, the model contains (i) a probabilistic encoder that maps from high-dimensional configuration space to a small-siz…

A neural network method estimates entropy production from system trajectories.

Paper learns meaningful state and action representations from MDP trajectories.

TD learning reduces prediction error in Markov chain problems.

The paper extends conformal prediction to MDP trajectories for autonomous systems.

Method estimates number of clusters in Block Markov Chain trajectories.

Policy analysts wish to visualize a range of policies for large simulator-defined Markov Decision Processes (MDPs). One visualization approach is to invoke the simulator to generate on-policy trajectories and then visualize those trajectories. When the simulator is expensive, this is not practical, and some method is r…

New algorithm clusters trajectories from multiple Markov chains with near-optimal error.

We study the problem of identity testing of markov chains. In this setting, we are given access to a single trajectory from a markov chain with unknown transition matrix $Q$ and the goal is to determine whether $Q = P$ for some known matrix $P$ or $\text{Dist}(P, Q) \geq ε$ where $\text{Dist}$ is suitably defined. In r…

Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the t…

The paper proposes a method to learn low-dimensional state embeddings from time series data.

Paper learns Markov models from data with low-rank optimization.

Infinite Hidden Markov Models (iHMM's) are an attractive, nonparametric generalization of the classical Hidden Markov Model which can automatically infer the number of hidden states in the system. However, due to the infinite-dimensional nature of transition dynamics performing inference in the iHMM is difficult. In th…

VLBM learns MDP transitions from limited data, improving OPE performance.

New insights on offline RL with state aggregation and trajectory data.

A new method simulates a lazy version of a Markov chain for empirical inference.

DDD reformulated for sparse matrices, integrating trajectory and snapshot time series data.

New method detects metastable basins in high dimensions using trajectory sampling.

This paper studies the estimation of low-rank Markov chains from empirical trajectories. We propose a non-convex estimator based on rank-constrained likelihood maximization. Statistical upper bounds are provided for the Kullback-Leiber divergence and the $\ell_2$ risk between the estimator and the true transition matri…

LSS learns molecular trajectories from MD data.

Modeling driver trajectories using inverse reinforcement learning and random utility.

Adaptive stopping in MCMC using classifier-based dynamics

Asynchronous Q-learning achieves optimal Q-function estimation with reduced sample complexity.

Markov jump processes (MJPs) are used to model a wide range of phenomena from disease progression to RNA path folding. However, maximum likelihood estimation of parametric models leads to degenerate trajectories and inferential performance is poor in nonparametric models. We take a small-variance asymptotics (SVA) appr…

Paper proposes a reinforcement learning method for trading using expert trajectories.

Neural models learn continuous-time Markov chain transition rates from data.

Many real-valued stochastic time-series are locally linear (Gassian), but globally non-linear. For example, the trajectory of a human hand gesture can be viewed as a linear dynamic system driven by a nonlinear dynamic system that represents muscle actions. We present a mixed-state dynamic graphical model in which a hid…

Study optimal offline RL with uncertainty sets and distribution shifts.

Dynamical systems with large state-spaces are often expensive to thoroughly explore experimentally. Coarse-graining methods aim to define simpler systems which are more amenable to analysis and exploration; most current methods, however, focus on a priori state aggregation based on similarities in transition rates, whi…

We present a data-efficient reinforcement learning algorithm resistant to observation noise. Our method extends the highly data-efficient PILCO algorithm (Deisenroth & Rasmussen, 2011) into partially observed Markov decision processes (POMDPs) by considering the filtering process during policy evaluation. PILCO conduct…

Schrödinger bridge solved with Weyl calculus for quadratic state cost.

Trajectory data suffices for efficient RL in linear MDPs.

This study models target trajectories using stochastic processes for efficient tracking.

Paper proposes method for optimal control of unknown systems with latent states.

Modeling continuous-time physiological processes that manifest a patient's evolving clinical states is a key step in approaching many problems in healthcare. In this paper, we develop the Hidden Absorbing Semi-Markov Model (HASMM): a versatile probabilistic model that is capable of capturing the modern electronic healt…

New algorithm reduces sample complexity for planning in MDPs.

Matrix Chernoff bound for Markov chains applied to co-occurrence matrices.

Reinforcement Learning optimizes low-thrust interplanetary trajectories under disturbances.

Particle Markov chain Monte Carlo (PMCMC) is a systematic way of combining the two main tools used for Monte Carlo statistical inference: sequential Monte Carlo (SMC) and Markov chain Monte Carlo (MCMC). We present a novel PMCMC algorithm that we refer to as particle Gibbs with ancestor sampling (PGAS). PGAS provides t…

This paper sets a lower bound for sample complexity in inverse reinforcement learning.

Estimates missing mass in Markovian sequences with linear runtime and near-optimal risk.

Particle Markov chain Monte Carlo techniques rank among current state-of-the-art methods for probabilistic program inference. A drawback of these techniques is that they rely on importance resampling, which results in degenerate particle trajectories and a low effective sample size for variables sampled early in a prog…

Study Markov chain gradient descent in Hilbert spaces for quadratic loss.