arXiv research

The paper simplifies multi-agent RL dynamics in finite-state Markov games using homogenization.

In his 2011 work, Maas has shown that the law of any time-reversible continuous-time Markov chain with finite state space evolves like a gradient flow of the relative entropy with respect to its stationary distribution. In this work we show the converse to the above by showing that if the relative law of a Markov chain…

We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an algorithm whose regret with respect to any policy in a comparison class grows as t…

Two neural network methods solve the master equation for MFGs.

A framework solves parametric families of MFGs efficiently.

Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the t…

New approach to understand recurrent policies as FSMs without minimization.

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using …

This paper develops a Hoeffding inequality for the partial sums $\sum_{k=1}^n f (X_k)$, where $\{X_k\}_{k \in \mathbb{Z}_{> 0}}$ is an irreducible Markov chain on a finite state space $S$, and $f : S \to [a, b]$ is a real-valued function. Our bound is simple, general, since it only assumes irreducibility and finiteness…

The potential approach is a general and simple method for modelling interest rates, foreign exchange rates, and in principle other types of financial assets. This paper takes data on some liquid interest rate derivatives, and fits potential models using a small finite-state Markov chain as the base Markov process.

Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.

In this paper we propose a semi-Markov modulated model of interest rates. We assume that the switching process is a semi-Markov process with finite state space E and the modulated process is a diffusive process. We derive recursive equations for the higher order moments of the discount factor and we describe a Monte Ca…

We study (backward) stochastic differential equations with noise coming from a finite state Markov chain. We show that, for the solutions of these equations to be `Markovian', in the sense that they are deterministic functions of the state of the underlying chain, the integrand must be of a specific form. This allows u…

New algorithms for RL in Markov games with independent linear function approximation, breaking the curse of multiagents.

A quantum financial approach to finite games of strategy is addressed, with an extension of Nash's theorem to the quantum financial setting, allowing for an entanglement of games of strategy with two-period financial allocation problems that are expressed in terms of: the consumption plans' optimization problem in pure…

We consider the estimation of the policy gradient in partially observable Markov decision processes (POMDP) with a special class of structured policies that are finite-state controllers. We show that the gradient estimation can be done in the Actor-Critic framework, by making the critic compute a "value" function that …

A new method uses Mean Field Games to optimize mixture models of Bernoulli and categorical distributions.

New assumptions and algorithm solve offline two-player zero-sum Markov games.

New algorithm finds near-optimal policies efficiently in zero-sum games.

We consider a self-exciting counting process, the parameters of which depend on a hidden finite-state Markov chain. We derive the optimal filter and smoother for the hidden chain based on observation of the jump process. This filter is in closed form and is finite dimensional. We demonstrate the performance of this fil…

Algorithm minimizes regret and converges to equilibria in Markov games.

Improved model-based reinforcement learning for multi-agent Markov games.

Study online learning in unknown Markov games with sublinear regret.

New algorithms achieve logarithmic regret in KL-regularized Markov games.

The existence of stationary Markov perfect equilibria in stochastic games is shown under a general condition called "(decomposable) coarser transition kernels". This result covers various earlier existence results on correlated equilibria, noisy stochastic games, stochastic games with finite actions and state-independe…

Algorithm finds ε-equilibrium policies for multi-agent Markov games with hidden low-rank structure.

The method approximates stationary distributions of Markov models by truncating irrelevant states.

Paper develops efficient algorithms for zero-sum Markov games with general function classes.

Paper optimizes multi-agent learning in Markov games with generative model.

New RL algorithms find SNE in Markov games with myopic followers.

Study efficient offline RL in Markov games with general models.

Researchers calculate Shannon entropy rates of hidden Markov processes efficiently.

Epsilon-machines are minimal, unifilar presentations of stationary stochastic processes. They were originally defined in the history machine sense, as hidden Markov models whose states are the equivalence classes of infinite pasts with the same probability distribution over futures. In analyzing synchronization, though…

Optimal algorithm for two-player zero-sum games with linear parameterization.

Pessimistic model-based algorithm finds Nash equilibria in zero-sum Markov games from offline data.

The paper analyzes Q-learning in 2-player Markov games and provides gap-dependent logarithmic regret bounds.

New research shows no-regret learning is impossible in Markov games under certain assumptions.

Develops CLTs for Markov chain transition probabilities and policies.

Efficient reinforcement learning for simultaneous-move zero-sum games using optimistic value iteration.

MF-TRPO optimizes MFGs with finite sample guarantees.

Study on natural actor-critic for POMDPs with finite memory.

Contributions: Prior studies on education have mostly followed the model of the cross sectional study, namely, examining the pretest and the posttest scores. This paper shows that students' knowledge throughout the intervention can be estimated by time series analysis using a hidden Markov model. Background: Analyzing …

This work tackles learning Markov games with adversarial opponents and achieves both average reward and exploitation.

A new method solves bilevel optimization problems in competitive Markov games.

Study on learning strategies in adaptive Markov games with policy regret as metric.

Study of multi-armed bandits with state-switching rewards using Markov models.

Algorithm finds Nash equilibria in complex games with function approximation.

New algorithm improves sample efficiency for zero-sum Markov games.