arXiv research

New approach to understand recurrent policies as FSMs without minimization.

We analyze how an observer synchronizes to the internal state of a finite-state information source, using the epsilon-machine causal representation. Here, we treat the case of exact synchronization, when it is possible for the observer to synchronize completely after a finite number of observations. The more difficult …

Study shows challenges in converting RNNs to FSMs due to computational complexity.

Epsilon-machines are minimal, unifilar presentations of stationary stochastic processes. They were originally defined in the history machine sense, as hidden Markov models whose states are the equivalence classes of infinite pasts with the same probability distribution over futures. In analyzing synchronization, though…

Transformers simulate finite-state automata with fewer layers.

Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the t…

New communication standards need to deal with machine-to-machine communications, in which users may start or stop transmitting at any time in an asynchronous manner. Thus, the number of users is an unknown and time-varying parameter that needs to be accurately estimated in order to properly recover the symbols transmit…

Recurrent neural networks trained on regular languages exhibit stable states that can recover from noise.

This work optimizes MCMC algorithms for modern accelerators without synchronization overheads.

The paper simplifies multi-agent RL dynamics in finite-state Markov games using homogenization.

A new layer learns abstract relations from graph structure using finite-state automata.

In this paper, we first establish the reflected backward stochastic difference equations with finite state (FS-RBSDEs for short). Then we explore the Existence and Uniqueness Theorem as well as the Comparison Theorem by "one step" method. The connections between FS-RBSDEs and optimal stopping time problems are investig…

Recurrent neural networks are a widely used class of neural architectures. They have, however, two shortcomings. First, it is difficult to understand what exactly they learn. Second, they tend to work poorly on sequences requiring long-term memorization, despite having this capacity in principle. We aim to address both…

In his 2011 work, Maas has shown that the law of any time-reversible continuous-time Markov chain with finite state space evolves like a gradient flow of the relative entropy with respect to its stationary distribution. In this work we show the converse to the above by showing that if the relative law of a Markov chain…

We extend a recent synchronization analysis of exact finite-state sources to nonexact sources for which synchronization occurs only asymptotically. Although the proof methods are quite different, the primary results remain the same. We find that an observer's average uncertainty in the source state vanishes exponential…

The paper explores geometric calculations on probability manifolds derived from master equations.

We propose an actor-critic, model-free, and online Reinforcement Learning (RL) framework for continuous-state continuous-action Markov Decision Processes (MDPs) when the reward is highly sparse but encompasses a high-level temporal structure. We represent this temporal structure by a finite-state machine and construct …

Framework for differentiating WFSTs for structured loss functions.

Two neural network methods solve the master equation for MFGs.

A framework solves parametric families of MFGs efficiently.

We generalize to the finite-state case the notion of the extreme effect variable $Y$ that accumulates all the effect of a variant variable $V$ observed in changes of another variable $X$. We conduct theoretical analysis and turn the problem of finding of an effect variable into a problem of a simultaneous decomposition…

This paper develops a Hoeffding inequality for the partial sums $\sum_{k=1}^n f (X_k)$, where $\{X_k\}_{k \in \mathbb{Z}_{> 0}}$ is an irreducible Markov chain on a finite state space $S$, and $f : S \to [a, b]$ is a real-valued function. Our bound is simple, general, since it only assumes irreducibility and finiteness…

The smallest eigenvectors of the graph Laplacian are well-known to provide a succinct representation of the geometry of a weighted graph. In reinforcement learning (RL), where the weighted graph may be interpreted as the state transition process induced by a behavior policy acting on the environment, approximating the …

Enhances reward specification in RL with a novel language-based approach.

Inverse reinforcement learning (IRL) is the problem of finding a reward function that generates a given optimal policy for a given Markov Decision Process. This paper looks at an algorithmic-independent geometric analysis of the IRL problem with finite states and actions. A L1-regularized Support Vector Machine formula…

This paper formalizes Uniswap v3 using PTA and FST for rigorous analysis.

The potential approach is a general and simple method for modelling interest rates, foreign exchange rates, and in principle other types of financial assets. This paper takes data on some liquid interest rate derivatives, and fits potential models using a small finite-state Markov chain as the base Markov process.

Develops STC for sequential data with missing labels.

The paper tackles restless bandits with limited observation, proposing a method to analyze and approximate their optimal strategies.

We study an open problem of risk-sensitive portfolio allocation in a regime-switching credit market with default contagion. The state space of the Markovian regime-switching process is assumed to be a countably infinite set. To characterize the value function, we investigate the corresponding recursive infinite-dimensi…

We consider the estimation of the policy gradient in partially observable Markov decision processes (POMDP) with a special class of structured policies that are finite-state controllers. We show that the gradient estimation can be done in the Actor-Critic framework, by making the critic compute a "value" function that …

We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montúfar and Ay,…

In this paper, a finite-state mean-reverting model for the short-rate, based on the continuous time Ehrenfest process, will be examined. Two explicit pricing formulae for zero-coupon bonds will be derived in the general and the special symmetric cases. Its limiting relationship to the Vasicek model will be examined wit…

Method infers causal structure from system behaviors using RKHS and kernel $ε$-machines.

We propose a new way of thinking about deep neural networks, in which the linear and non-linear components of the network are naturally derived and justified in terms of principles in probability theory. In particular, the models constructed in our framework assign probabilities to uncertain realizations, leading to Ku…

Reservoir computers (RCs) and recurrent neural networks (RNNs) can mimic any finite-state automaton in theory, and some workers demonstrated that this can hold in practice. We test the capability of generalized linear models, RCs, and Long Short-Term Memory (LSTM) RNN architectures to predict the stochastic processes g…

Study on natural actor-critic for POMDPs with finite memory.

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using …

We solve POMDPs by approximating them as finite-state MDPs.

Study growth patterns in random networks using i.i.d. perturbations.

We investigate the problem of reliable communication between two legitimate parties over deletion channels under an active eavesdropping (aka jamming) adversarial model. To this goal, we develop a theoretical framework based on probabilistic finite-state automata to define novel encoding and decoding schemes that ensur…

We consider a self-exciting counting process, the parameters of which depend on a hidden finite-state Markov chain. We derive the optimal filter and smoother for the hidden chain based on observation of the jump process. This filter is in closed form and is finite dimensional. We demonstrate the performance of this fil…

This review explores entropy applications in data analysis and machine learning.

Reciprocal processes are acausal generalizations of Markov processes introduced by Bernstein in 1932. In the literature, a significant amount of attention has been focused on developing dynamical models for reciprocal processes. In this paper, we provide a probabilistic graphical model for reciprocal processes. This le…

We study (backward) stochastic differential equations with noise coming from a finite state Markov chain. We show that, for the solutions of these equations to be `Markovian', in the sense that they are deterministic functions of the state of the underlying chain, the integrand must be of a specific form. This allows u…

In this paper we propose a semi-Markov modulated model of interest rates. We assume that the switching process is a semi-Markov process with finite state space E and the modulated process is a diffusive process. We derive recursive equations for the higher order moments of the discount factor and we describe a Monte Ca…

Multi-dimensional state-integrals of products of Faddeev's quantum dilogarithms arise frequently in Quantum Topology, quantum Teichmüller theory and complex Chern--Simons theory. Using the quasi-periodicity property of the quantum dilogarithm, we evaluate 1-dimensional state-integrals at rational points and express the…

We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structu…