arXiv research

New neural processes use stacked Markov operators to improve flexibility.

Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distributio…

In most sampling algorithms, including Hamiltonian Monte Carlo, transition rates between states correspond to the probability of making a transition in a single time step, and are constrained to be less than or equal to 1. We derive a Hamiltonian Monte Carlo algorithm using a continuous time Markov jump process, and ar…

We propose a novel method to directly learn a stochastic transition operator whose repeated application provides generated samples. Traditional undirected graphical models approach this problem indirectly by learning a Markov chain model whose stationary distribution obeys detailed balance with respect to a parameteriz…

Neural models learn continuous-time Markov chain transition rates from data.

Neural networks parameterize time-varying Markov dynamics in financial time series.

Bootstrap method for Markov chains in reinforcement learning.

A framework combining HSMM and survival analysis for lifecycle-oriented mobility analysis.

Develops CLTs for Markov chain transition probabilities and policies.

We derive the exact solution of a one-dimensional Markov functional model with log-normally distributed interest rates in discrete time. The model is shown to have two distinct limiting states, corresponding to small and asymptotically large volatilities, respectively. These volatility regimes are separated by a phase …

The paper proposes a method to improve Koopman operator estimation using indicator functions.

A new algorithm for deep Q-learning with robustness to state transition uncertainty.

A multi-task GP model tracks time-varying transition probabilities between two states.

Develops a new framework for conditional independence.

Researchers approximate conditional expectation operators using kernel methods.

Extended elliptical slice sampling for infinite-dimensional spaces, proving reversibility.

We describe a generalization of the Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM) which is able to encode prior information that state transitions are more likely between "nearby" states. This is accomplished by defining a similarity function on the state space and scaling transition probabilities by pai…

Graphical models are popular statistical tools which are used to represent dependent or causal complex systems. Statistically equivalent causal or directed graphical models are said to belong to a Markov equivalent class. It is of great interest to describe and understand the space of such classes. However, with curren…

The paper tackles data-driven optimal control of unknown nonlinear systems using RKHS.

This paper proposes a stochastic model using the concept of Markov chains for the inter-state transitions of the millisecond order quasi-stable phase synchronized patterns or synchrostates, found in multi-channel Electroencephalogram (EEG) signals. First and second order transition probability matrices are estimated fo…

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

The paper studies convergence of kernel autocovariance operators for stationary processes.

New insights into Markov chain geometry via positive transition measures.

Fluid approximations have seen great success in approximating the macro-scale behaviour of Markov systems with a large number of discrete states. However, these methods rely on the continuous-time Markov chain (CTMC) having a particular population structure which suggests a natural continuous state-space endowed with a…

Kernel-based methods improve policy evaluation in MRP models.

Deriving option prices from operational-time Markov lattices

We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an algorithm whose regret with respect to any policy in a comparison class grows as t…

New algorithms solve robust MDPs efficiently, significantly faster than existing methods.

Proves error bounds for state representation in RL using graph spectral features.

Free lunch from noise reveals linear spectral features for RL.

In this work, we investigate a novel training procedure to learn a generative model as the transition operator of a Markov chain, such that, when applied repeatedly on an unstructured random noise sample, it will denoise it into a sample that matches the target distribution from the training set. The novel training pro…

New algorithms achieve no-regret learning even with adversarial transitions and losses.

Reinforcement learning agents have been mostly developed and evaluated under the assumption that they will operate in a fully autonomous manner -- they will take all actions. In this work, our goal is to develop algorithms that, by learning to switch control between agents, allow existing reinforcement learning agents …

We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains sequentially to observe their states. The goal of the learner is to sequentially…

Identity testing for reversible Markov chains without symmetry assumption.

We consider reinforcement learning in changing Markov Decision Processes where both the state-transition probabilities and the reward functions may vary over time. For this problem setting, we propose an algorithm using a sliding window approach and provide performance guarantees for the regret evaluated against the op…

Faster sampling in discrete diffusion models with predetermined transition time.

One-step Bellman alignment improves online RL by reducing task mismatch.

New method detects changes in high-dimensional Markov processes without explicit likelihood evaluation.

Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…

Develops a flexible model for regime transitions in time series data.

Modified Metropolis algorithm ensures convergence for multivariate binary distributions with fixed-order updates.

New algorithms reduce regret in reinforcement learning with MNL approximations.

Paper proposes a new method for training diffusion models using Markov operators.

New method trains Markov kernels for efficient sampling.

TMTF improves time series visualization by separating dynamic regimes.

Algorithm finds ε-equilibrium policies for multi-agent Markov games with hidden low-rank structure.

Proposes MIVI for efficient posterior estimation and design of MCMC transitions.