New neural processes use stacked Markov operators to improve flexibility.
problem Improving flexibility in neural processes.
method Stacking neural parameterized Markov transition operators in function space.
result MNPs outperform baseline models on various tasks.
Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distributio…
In most sampling algorithms, including Hamiltonian Monte Carlo, transition rates between states correspond to the probability of making a transition in a single time step, and are constrained to be less than or equal to 1. We derive a Hamiltonian Monte Carlo algorithm using a continuous time Markov jump process, and ar…
Neural networks parameterize time-varying Markov dynamics in financial time series.
problem Estimating Markov transition matrices in high-resolution, high-noise financial data.
method Introduces a neural network framework to generate explicit, time-varying Markov transition matrices, constraining neural outputs to formal stochastic operators.
result Learned operators capture regime shifts, with high-volatility regimes homogenizing transition dynamics.
We propose a novel method to directly learn a stochastic transition operator whose repeated application provides generated samples. Traditional undirected graphical models approach this problem indirectly by learning a Markov chain model whose stationary distribution obeys detailed balance with respect to a parameteriz…
A new algorithm for deep Q-learning with robustness to state transition uncertainty.
problem Model uncertainty in state transitions for non-tabular, continuous state spaces.
method Distributionally robust approach using worst-case transition ball and dualized Bellman operator with Sinkhorn distance.
result Optimal policy found through solving non-linear Bellman equation with neural network parameterization.
Researchers approximate conditional expectation operators using kernel methods.
problem Statistical approximation of conditional expectation operators under minimal assumptions.
method Modifying the domain of the operator, approximating it by Hilbert-Schmidt operators in a reproducing kernel Hilbert space.
result The nonparametric estimate of the operator converges to a specific limiting object.
Extended elliptical slice sampling for infinite-dimensional spaces, proving reversibility.
problem Proving reversibility of elliptical slice sampling in infinite-dimensional spaces.
method Extended elliptical slice sampling to infinite-dimensional separable Hilbert spaces, providing an alternative proof of reversibility.
result The approach yields a positive semi-definite Markov operator, proving reversibility.
Graphical models are popular statistical tools which are used to represent dependent or causal complex systems. Statistically equivalent causal or directed graphical models are said to belong to a Markov equivalent class. It is of great interest to describe and understand the space of such classes. However, with curren…
A framework combining HSMM and survival analysis for lifecycle-oriented mobility analysis.
problem Understanding individual metro usage dynamics over multi-year horizons.
method A state-based lifecycle modeling framework integrating HSMM and discrete-time survival analysis.
result Identification of interpretable mobility states, transition dynamics, and state-dependent exit and re-entry processes.
This paper proposes a stochastic model using the concept of Markov chains for the inter-state transitions of the millisecond order quasi-stable phase synchronized patterns or synchrostates, found in multi-channel Electroencephalogram (EEG) signals. First and second order transition probability matrices are estimated fo…
Bootstrap method for Markov chains in reinforcement learning.
problem Distributional consistency in finite controlled Markov chains with unknown control policies.
method Model-based bootstrap with novel LLN and CLT for visitation counts and transition increments.
result Asymptotically valid confidence intervals for value and Q-functions in offline RL. The paper studies convergence of kernel autocovariance operators for stationary processes.
problem Estimating autocovariance operators of stationary processes on Polish spaces.
method Investigates convergence of empirical estimates of autocovariance operators under various conditions.
result Provides consistency results for kernel PCA and spectral analysis methods.
New insights into Markov chain geometry via positive transition measures.
problem Lack of statistical meaning in the space of transition probabilities.
method Constructing an extension of the space of transition probabilities using Amari's theory of positive measures.
result Introduction of a new dually flat structure for the space of positive transition measures.
Fluid approximations have seen great success in approximating the macro-scale behaviour of Markov systems with a large number of discrete states. However, these methods rely on the continuous-time Markov chain (CTMC) having a particular population structure which suggests a natural continuous state-space endowed with a…
Algorithm learns to switch control among agents in a team.
problem Learning to switch control among reinforcement learning agents.
method 2-layer Markov decision process, upper confidence bounds, shared confidence bounds.
result Sublinear total regret with shared confidence bounds.
Develops CLTs for Markov chain transition probabilities and policies.
problem Estimating transition probabilities and policies in controlled Markov chains.
method Non-parametric estimator for transition matrices; CLTs for value, Q-, and advantage functions; goodness-of-fit tests.
result Asymptotic normality of estimators under specific logging policies.
Neural models learn continuous-time Markov chain transition rates from data.
problem Learning transition rates for complex stochastic systems.
method Neural networks to model nonlinear transition rates from observed data.
result Neural models outperform traditional methods in accuracy.
Deriving option prices from operational-time Markov lattices
problem Option pricing
method Operational-time Markov lattice
result Derives option-pricing equations from an operational-time Markov lattice
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
Free lunch from noise reveals linear spectral features for RL.
problem Trade-off between expressiveness and tractability in RL.
method Noise assumption and Spectral Dynamics Embedding (SPEDE).
result SPEDE breaks the trade-off and completes optimistic exploration.
In this work, we investigate a novel training procedure to learn a generative model as the transition operator of a Markov chain, such that, when applied repeatedly on an unstructured random noise sample, it will denoise it into a sample that matches the target distribution from the training set. The novel training pro…
Develops a new framework for conditional independence.
problem Generalizing previous notions of conditional independence.
method Introduces transition probability spaces and transitional random variables.
result Satisfies all desired relevance relations except symmetry.
Identity testing for reversible Markov chains without symmetry assumption.
problem Identity testing of reversible Markov chains.
method Using distance notion from Daskalakis et al. [2018a], testing without symmetry assumption.
result It is possible to perform identity testing under weaker assumption of reversibility.
New method detects changes in high-dimensional Markov processes without explicit likelihood evaluation.
problem Quickest change detection in Markov processes with unknown transition kernels.
method Learn conditional score from sample pairs, develop score-based CUSUM procedure.
result Exponential lower bounds on mean time to false alarm and asymptotic upper bounds on detection delay.
Modified Metropolis algorithm ensures convergence for multivariate binary distributions with fixed-order updates.
problem Infeasibility of standard Metropolis algorithm for multivariate binary distributions with fixed-order updates.
method Proposed a modified Metropolis transition operator ensuring irreducibility and convergence.
result Ensures convergence to the limiting distribution in multivariate binary case with fixed-order updates.
TMTF improves time series visualization by separating dynamic regimes.
problem Misleading global transition matrix in time series analysis.
method Temporal chunking, local transition matrices, and image assembly.
result Temporal segmentation reveals distinct transition dynamics.
Proposes MIVI for efficient posterior estimation and design of MCMC transitions.
problem Efficiently estimating posterior distributions in constrained time.
method Combines variational inference and MCMC with a variational distribution and optimized Markov chain.
result Optimized Markov chain improves variational distribution and vice versa, leading to more accurate posteriors.
Paper tackles matrix estimation under arbitrary noise, achieving minimax optimality.
problem Noisy low-rank-plus-sparse matrix recovery under arbitrary dependence.
method Incoherent-constrained least-square estimator, novel energy spreading result.
result Achieves minimax optimality in estimating structured Markov transition kernels.
A multi-task GP model tracks time-varying transition probabilities between two states.
problem Tracking time-varying transition probabilities between 'moves' and 'pauses' states.
method Kernel-based multi-task Gaussian Process model with time-variability and constraints.
result Enforces constraints while learning transition probabilities.
Two-dimensional transition rates improve life insurance reserve calculations.
problem Calculating life insurance reserves with Markov assumptions.
method Introducing two-dimensional forward and backward transition rates.
result Two-dimensional transition rates enable more accurate reserve calculations.
Expands Hidden Markov Model to include Markov chain observations.
problem Handling Markov chain observations in Hidden Markov Models.
method Developed Expectation-Maximization algorithm and Viterbi algorithm analogs.
result Estimates transition probabilities for hidden states and observations.
Novel unsupervised feature selection method using multi-step Markov transition probability.
problem Neglected relationships between non-adjacent data points in feature selection.
method MMFS (Multi-step Markov transition probability for Feature Selection) approach, employing positive and negative viewpoints.
result MMFS effectively maintains data structure in unsupervised feature selection.
We introduce a new geometric approach that constructs a transition kernel of Markov chain. Our method always minimizes the average rejection rate and even reduce it to zero in many relevant cases, which cannot be achieved by conventional methods, such as the Metropolis-Hastings algorithm or the heat bath algorithm (Gib…
A Semi-Hidden Markov Model (SHMM) for bursty error channels is defined by a state transition probability matrix A, a prior probability vector Π, and the state dependent output symbol error probability matrix B. Several processes are utilized for estimating A, Π and B from a given empirically obtained or sim…
We describe a generalization of the Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM) which is able to encode prior information that state transitions are more likely between "nearby" states. This is accomplished by defining a similarity function on the state space and scaling transition probabilities by pai…
We derive the exact solution of a one-dimensional Markov functional model with log-normally distributed interest rates in discrete time. The model is shown to have two distinct limiting states, corresponding to small and asymptotically large volatilities, respectively. These volatility regimes are separated by a phase …
The paper proposes a method to improve Koopman operator estimation using indicator functions.
problem Difficulty in identifying good observables for Koopman operator expansion.
method Clustering procedure based on Hidden Markov Model (HMM) to infer surrogate observables.
result Inferred indicator functions significantly improve estimation of Koopman operator eigenvalues and transition timescales.
Model reduction of Markov processes is a basic problem in modeling state-transition systems. Motivated by the state aggregation approach rooted in control theory, we study the statistical state compression of a discrete-state Markov chain from empirical trajectories. Through the lens of spectral decomposition, we study…
New issue found in value-based reinforcement learning for stochastic environments.
problem Value-based reinforcement learning struggles with stochastic state transitions.
method Demonstrated using a multiobjective Markov Decision Process (MOMDP).
result Approaches may converge to Pareto-dominated solutions instead of optimal ones.
Study nonparametric estimator for Markov chain transition matrices in offline setting.
problem Estimating transition matrices of finite controlled Markov chains from logged data.
method Developed sample complexity bounds and conditions for minimaxity.
result Achieving certain statistical risk requires balancing mixing properties and sample size.
Proves error bounds for state representation in RL using graph spectral features.
problem Addressing the curse of dimensionality in RL with unknown transition graphs.
method Proves upper bounds on approximation error of linear value function approximation using learned spectral features of the state-graph.
result Error bounds scale with algebraic connectivity and eigenvector estimation error.
A new model separates persistence and transition priors in HDP-HMM.
problem Limitation of sticky HDP-HMM in expressing different persistence strengths.
method Developed a disentangled sticky HDP-HMM (DS-HDP-HMM) with novel Gibbs sampling algorithms.
result DS-HDP-HMM outperforms sticky HDP-HMM and HDP-HMM on synthetic and real data.
The existence of stationary Markov perfect equilibria in stochastic games is shown under a general condition called "(decomposable) coarser transition kernels". This result covers various earlier existence results on correlated equilibria, noisy stochastic games, stochastic games with finite actions and state-independe…
We search for digital biomarkers from Parkinson's Disease by observing approximate repetitive patterns matching hypothesized step and stride periodic cycles. These observations were modeled as a cycle of hidden states with randomness allowing deviation from a canonical pattern of transitions and emissions, under the hy…
We review some developments concerning Markov and Feller processes with jumps in geometric settings. These include stochastic differential equations in Markus canonical form, the Courrège theorem on Lie groups, and invariant Markov processes on manifolds under both transitive and more general Lie group actions.
New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.
problem Achieving best-of-both-worlds guarantees with data-dependent regret bounds in MDPs with unknown transitions.
method Optimistic follow-the-regularized-leader algorithm with new optimistic Q-function estimators and transition bonus.
result First-order, second-order, and path-length bounds with polylog(T) regret in the stochastic regime.
Study analyzes order transitions in high, medium, and low market cap stocks using Markov chains.
problem Understanding order transitions in stocks of different market caps.
method First-order discrete-time Markov chain model applied to NASDAQ100 stocks.
result Limit orders exhibit higher inertia during opening hours but decrease in subsequent hours, while market orders increase.