arXiv research

New neural processes use stacked Markov operators to improve flexibility.

This paper proposes a stochastic model using the concept of Markov chains for the inter-state transitions of the millisecond order quasi-stable phase synchronized patterns or synchrostates, found in multi-channel Electroencephalogram (EEG) signals. First and second order transition probability matrices are estimated fo…

New insights into Markov chain geometry via positive transition measures.

Develops CLTs for Markov chain transition probabilities and policies.

Neural models learn continuous-time Markov chain transition rates from data.

Develops a new framework for conditional independence.

New method detects changes in high-dimensional Markov processes without explicit likelihood evaluation.

In most sampling algorithms, including Hamiltonian Monte Carlo, transition rates between states correspond to the probability of making a transition in a single time step, and are constrained to be less than or equal to 1. We derive a Hamiltonian Monte Carlo algorithm using a continuous time Markov jump process, and ar…

TMTF improves time series visualization by separating dynamic regimes.

Proposes MIVI for efficient posterior estimation and design of MCMC transitions.

A multi-task GP model tracks time-varying transition probabilities between two states.

Two-dimensional transition rates improve life insurance reserve calculations.

Expands Hidden Markov Model to include Markov chain observations.

Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distributio…

Novel unsupervised feature selection method using multi-step Markov transition probability.

We introduce a new geometric approach that constructs a transition kernel of Markov chain. Our method always minimizes the average rejection rate and even reduce it to zero in many relevant cases, which cannot be achieved by conventional methods, such as the Metropolis-Hastings algorithm or the heat bath algorithm (Gib…

A Semi-Hidden Markov Model (SHMM) for bursty error channels is defined by a state transition probability matrix $A$, a prior probability vector $Π$, and the state dependent output symbol error probability matrix $B$. Several processes are utilized for estimating $A$, $Π$ and $B$ from a given empirically obtained or sim…

We describe a generalization of the Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM) which is able to encode prior information that state transitions are more likely between "nearby" states. This is accomplished by defining a similarity function on the state space and scaling transition probabilities by pai…

We derive the exact solution of a one-dimensional Markov functional model with log-normally distributed interest rates in discrete time. The model is shown to have two distinct limiting states, corresponding to small and asymptotically large volatilities, respectively. These volatility regimes are separated by a phase …

Model reduction of Markov processes is a basic problem in modeling state-transition systems. Motivated by the state aggregation approach rooted in control theory, we study the statistical state compression of a discrete-state Markov chain from empirical trajectories. Through the lens of spectral decomposition, we study…

New issue found in value-based reinforcement learning for stochastic environments.

Study nonparametric estimator for Markov chain transition matrices in offline setting.

A new model separates persistence and transition priors in HDP-HMM.

The existence of stationary Markov perfect equilibria in stochastic games is shown under a general condition called "(decomposable) coarser transition kernels". This result covers various earlier existence results on correlated equilibria, noisy stochastic games, stochastic games with finite actions and state-independe…

We search for digital biomarkers from Parkinson's Disease by observing approximate repetitive patterns matching hypothesized step and stride periodic cycles. These observations were modeled as a cycle of hidden states with randomness allowing deviation from a canonical pattern of transitions and emissions, under the hy…

We review some developments concerning Markov and Feller processes with jumps in geometric settings. These include stochastic differential equations in Markus canonical form, the Courrège theorem on Lie groups, and invariant Markov processes on manifolds under both transitive and more general Lie group actions.

A new algorithm for deep Q-learning with robustness to state transition uncertainty.

New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.

Study analyzes order transitions in high, medium, and low market cap stocks using Markov chains.

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using …

We consider the problem of estimating the transition rate matrix of a continuous-time Markov chain from a finite-duration realisation of this process. We approach this problem in an imprecise probabilistic framework, using a set of prior distributions on the unknown transition rate matrix. The resulting estimator is a …

Optimal sequential testing for Markovian data with lower and upper bounds.

We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an algorithm whose regret with respect to any policy in a comparison class grows as t…

A framework combining HSMM and survival analysis for lifecycle-oriented mobility analysis.

We study the problem of learning the transition matrices of a set of Markov chains from a single stream of observations on each chain. We assume that the Markov chains are ergodic but otherwise unknown. The learner can sample Markov chains sequentially to observe their states. The goal of the learner is to sequentially…

Study analyzes stock order transitions during US-China trade war using Markov chains.

Faster sampling in discrete diffusion models with predetermined transition time.

Neural Markov models improve time series analysis by balancing deep learning and classical models.

Bootstrap method for Markov chains in reinforcement learning.

Estimates bisimulation metrics from sample streams, not full transition models.

GLASS Flows improves flow and diffusion model performance by optimizing sampling efficiency.

We consider reinforcement learning in changing Markov Decision Processes where both the state-transition probabilities and the reward functions may vary over time. For this problem setting, we propose an algorithm using a sliding window approach and provide performance guarantees for the regret evaluated against the op…

A low-rank tensor model simplifies multi-dimensional Markov chains.

Study analyzes price change patterns across different market capitalizations using Markov chains.

We propose a Markov chain model for credit rating changes. We do not use any distributional assumptions on the asset values of the rated companies but directly model the rating transitions process. The parameters of the model are estimated by a maximum likelihood approach using historical rating transitions and heurist…

Deep reinforcement learning finds optimal learning policies for adaptive systems.

We define a Hidden Markov Model (HMM) in which each hidden state has time-dependent $\textit{activity levels}$ that drive transitions and emissions, and show how to estimate its parameters. Our construction is motivated by the problem of inferring human mobility on sub-daily time scales from, for example, mobile phone …

Paper tackles identifying an odd arm in a multi-armed bandit with restless Markov processes and trembling hand.