arXiv research

A new method simulates a lazy version of a Markov chain for empirical inference.

The paper tackles learning from non-irreducible Markov chains, proving learnability and generalization bounds.

The paper develops a Hoeffding inequality for Markov chains and applies it to bandit problems.

The paper analyzes learning rates for non-irreducible Markov chains.

Modified Metropolis algorithm ensures convergence for multivariate binary distributions with fixed-order updates.

Matrix Chernoff bound for Markov chains applied to co-occurrence matrices.

Let K be an irreducible and reversible Markov kernel on a finite set X. We construct a metric W on the set of probability measures on X and show that with respect to this metric, the law of the continuous time Markov chain evolves as the gradient flow of the entropy. This result is a discrete counterpart of the Wassers…

We consider the restless Markov bandit problem, in which the state of each arm evolves according to a Markov process independently of the learner's actions. We suggest an algorithm that after $T$ steps achieves $\tilde{O}(\sqrt{T})$ regret with respect to the best policy that knows the distributions of all arms. No ass…

Graphical models are popular statistical tools which are used to represent dependent or causal complex systems. Statistically equivalent causal or directed graphical models are said to belong to a Markov equivalent class. It is of great interest to describe and understand the space of such classes. However, with curren…

We consider the problem of estimating from sample paths the absolute spectral gap $γ_*$ of a reversible, irreducible and aperiodic Markov chain $(X_t)_{t \in \mathbb{N}}$ over a finite state space $Ω$. We propose the ${\tt UCPI}$ (Upper Confidence Power Iteration) algorithm for this problem, a low-complexity algorithm …

Paper improves Oja's algorithm for Markovian data streams.

TSAW improves MCMC integral estimation with faster convergence.

Minimal assumptions analysis of Q-learning with time-varying policies.

Paper proves CLT for quantile SGD with constant learning rate.

The paper provides concentration inequalities for Markov chain variance estimators.

Reduces identity testing of reversible Markov chains to simpler symmetric chain tests.

Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the t…

Study Markov chain gradient descent in Hilbert spaces for quadratic loss.

Study nonparametric estimator for Markov chain transition matrices in offline setting.

Enhanced Markov chain sampler learns network statistics faster.

Stochastic kernel based dimensionality reduction approaches have become popular in the last decade. The central component of many of these methods is a symmetric kernel that quantifies the vicinity between pairs of data points and a kernel-induced Markov chain on the data. Typically, the Markov chain is fully specified…

Expands Hidden Markov Model to include Markov chain observations.

In this paper we describe three stochastic models based on a semi-Markov chains approach and its generalizations to study the high frequency price dynamics of traded stocks. The three models are: a simple semi-Markov chain model, an indexed semi-Markov chain model and a weighted indexed semi-Markov chain model. We show…

New insights into Markov chain geometry via positive transition measures.

In his 2011 work, Maas has shown that the law of any time-reversible continuous-time Markov chain with finite state space evolves like a gradient flow of the relative entropy with respect to its stationary distribution. In this work we show the converse to the above by showing that if the relative law of a Markov chain…

New method estimates convergence bounds for nonlinear Markov chains.

Elo ratings learn model parameters quickly using Markov chains.

DCDC calculates convergence rates for Markov chains using neural networks.

The paper studies how quickly samples from Langevin dynamics become independent.

We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains sequentially to observe their states. The goal of the learner is to sequentially…

This paper models time-series data with a mixture of Markov chains, automatically determining the number of components.

Method reconstructs hidden Markov chains from insurance data.

The paper extends Hoeffding's inequality for Markov chains using a generalized concentrability condition.

Identity testing for reversible Markov chains without symmetry assumption.

Unbiased gradient estimation for Markov chains

The method of block coordinate gradient descent (BCD) has been a powerful method for large-scale optimization. This paper considers the BCD method that successively updates a series of blocks selected according to a Markov chain. This kind of block selection is neither i.i.d. random nor cyclic. On the other hand, it is…

New framework improves variational inference with Markov chain methods.

The paper develops new inequalities for Markov chain sums, linking them to mixing time.

This paper proposes a stochastic model using the concept of Markov chains for the inter-state transitions of the millisecond order quasi-stable phase synchronized patterns or synchrostates, found in multi-channel Electroencephalogram (EEG) signals. First and second order transition probability matrices are estimated fo…

DMGD algorithm solves large-scale ML problems in costly sampling settings.

The time to converge to the steady state of a finite Markov chain can be greatly reduced by a lifting operation, which creates a new Markov chain on an expanded state space. For a class of quadratic objectives, we show an analogous behavior where a distributed ADMM algorithm can be seen as a lifting of Gradient Descent…

Policy gradient algorithm with variable learning rates achieves near-optimal performance in multi-arm bandit problems.

Algorithm learns mixtures of Markov chains and MDPs from short trajectories.

We study the problem of identity testing of markov chains. In this setting, we are given access to a single trajectory from a markov chain with unknown transition matrix $Q$ and the goal is to determine whether $Q = P$ for some known matrix $P$ or $\text{Dist}(P, Q) \geq ε$ where $\text{Dist}$ is suitably defined. In r…

We study (backward) stochastic differential equations with noise coming from a finite state Markov chain. We show that, for the solutions of these equations to be `Markovian', in the sense that they are deterministic functions of the state of the underlying chain, the integrand must be of a specific form. This allows u…

Non-negative curvature affects Markov chains' mixing and expansion properties.

Paper proposes semi-supervised learning with triplet Markov chains.

We present a new family of models that is based on graphs that may have undirected, directed and bidirected edges. We name these new models marginal AMP (MAMP) chain graphs because each of them is Markov equivalent to some AMP chain graph under marginalization of some of its nodes. However, MAMP chain graphs do not onl…