arXiv research

Expands Hidden Markov Model to include Markov chain observations.

In this paper we discuss four problems regarding Markov equivalences for subclasses of loopless mixed graphs. We classify these four problems as finding conditions for internal Markov equivalence, which is Markov equivalence within a subclass, for external Markov equivalence, which is Markov equivalence between subclas…

New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.

Improved model-based reinforcement learning for multi-agent Markov games.

Algorithm learns mixtures of Markov chains and MDPs from short trajectories.

New algorithms for RL in Markov games with independent linear function approximation, breaking the curse of multiagents.

New algorithm learns causal structures by intersecting Markov blankets.

New assumptions and algorithm solve offline two-player zero-sum Markov games.

Policy gradient algorithm with variable learning rates achieves near-optimal performance in multi-arm bandit problems.

Study Markov chain gradient descent in Hilbert spaces for quadratic loss.

Unified framework for drawdown risk computation under Markov models.

New entropy flow method extends generalization bounds for all Markov algorithms.

New algorithm for collective Gaussian hidden Markov models inference.

The paper estimates key metrics for linear models with Markov or hidden Markov sources.

In this paper we present an algorithm for pricing barrier options in one-dimensional Markov models. The approach rests on the construction of an approximating continuous-time Markov chain that closely follows the dynamics of the given Markov model. We illustrate the method by implementing it for a range of models, incl…

Stochastic variational inference for collapsed models has recently been successfully applied to large scale topic modelling. In this paper, we propose a stochastic collapsed variational inference algorithm for hidden Markov models, in a sequential data setting. Given a collapsed hidden Markov Model, we break its long M…

Paper develops efficient algorithms for zero-sum Markov games with general function classes.

Study on gradient descent in Hilbert spaces with Markov chains, focusing on mixing coefficients.

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

Decentralized stochastic gradient method emerges as a promising solution for solving large-scale machine learning problems. This paper studies the decentralized Markov chain gradient descent (DMGD) algorithm - a variant of the decentralized stochastic gradient methods where the random samples are taken along the trajec…

New algorithm finds near-optimal policies efficiently in zero-sum games.

New algorithm detects changes in Markov kernels with unknown post-change kernel.

In most sampling algorithms, including Hamiltonian Monte Carlo, transition rates between states correspond to the probability of making a transition in a single time step, and are constrained to be less than or equal to 1. We derive a Hamiltonian Monte Carlo algorithm using a continuous time Markov jump process, and ar…

New algorithm processes Riemannian data more efficiently.

New algorithm improves volatility forecasting using Pairwise Markov Chains.

Polynomial-time methods count and sample DAGs from Markov classes.

This is a technical report which explores the estimation methodologies on hyper-parameters in Markov Random Field and Gaussian Hidden Markov Random Field. In first section, we briefly investigate a theoretical framework on Metropolis-Hastings algorithm. Next, by using MH algorithm, we simulate the data from Ising model…

DCDC calculates convergence rates for Markov chains using neural networks.

Algorithm minimizes regret and converges to equilibria in Markov games.

A simple linear algebraic explanation of the algorithm in "A Spectral Algorithm for Learning Hidden Markov Models" (COLT 2009). Most of the content is in Figure 2; the text just makes everything precise in four nearly-trivial claims.

New algorithm improves sample efficiency for zero-sum Markov games.

Paper presents an algorithm for optimal regret in communicating Markov decision processes.

In reinforcement learning (RL) , one of the key components is policy evaluation, which aims to estimate the value function (i.e., expected long-term accumulated reward) of a policy. With a good policy evaluation method, the RL algorithms will estimate the value function more accurately and find a better policy. When th…

Polynomial-time methods count and sample DAGs from equivalence classes.

The time to converge to the steady state of a finite Markov chain can be greatly reduced by a lifting operation, which creates a new Markov chain on an expanded state space. For a class of quadratic objectives, we show an analogous behavior where a distributed ADMM algorithm can be seen as a lifting of Gradient Descent…

Algorithm finds ε-equilibrium policies for multi-agent Markov games with hidden low-rank structure.

Given a Gaussian Markov random field, we consider the problem of selecting a subset of variables to observe which minimizes the total expected squared prediction error of the unobserved variables. We first show that finding an exact solution is NP-hard even for a restricted class of Gaussian Markov random fields, calle…

New MCMC method for complex models with large variables.

Unified framework for analyzing convergence of RSAs using Wasserstein divergence.

Study of multi-armed bandits with state-switching rewards using Markov models.

We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains sequentially to observe their states. The goal of the learner is to sequentially…

New algorithm clusters trajectories from multiple Markov chains with near-optimal error.

We study the problem of identity testing of markov chains. In this setting, we are given access to a single trajectory from a markov chain with unknown transition matrix $Q$ and the goal is to determine whether $Q = P$ for some known matrix $P$ or $\text{Dist}(P, Q) \geq ε$ where $\text{Dist}$ is suitably defined. In r…

This thesis classifies pseudo-Anosov homeomorphisms using geometric Markov partitions.

Graphical models are popular statistical tools which are used to represent dependent or causal complex systems. Statistically equivalent causal or directed graphical models are said to belong to a Markov equivalent class. It is of great interest to describe and understand the space of such classes. However, with curren…

Gaussian Belief Propagation (BP) algorithm is one of the most important distributed algorithms in signal processing and statistical learning involving Markov networks. It is well known that the algorithm correctly computes marginal density functions from a high dimensional joint density function over a Markov network i…

Algorithm selects optimal experiments in Markov chains to learn unknown quantities.

In this paper, we propose a simple, versatile model for learning the structure and parameters of multivariate distributions from a data set. Learning a Markov network from a given data set is not a simple problem, because Markov networks rigorously represent Markov properties, and this rigor imposes complex constraints…