New method detects changes in high-dimensional Markov processes without explicit likelihood evaluation.
problem Quickest change detection in Markov processes with unknown transition kernels.
method Learn conditional score from sample pairs, develop score-based CUSUM procedure.
result Exponential lower bounds on mean time to false alarm and asymptotic upper bounds on detection delay.
We introduce a new geometric approach that constructs a transition kernel of Markov chain. Our method always minimizes the average rejection rate and even reduce it to zero in many relevant cases, which cannot be achieved by conventional methods, such as the Metropolis-Hastings algorithm or the heat bath algorithm (Gib…
Develops a new framework for conditional independence.
problem Generalizing previous notions of conditional independence.
method Introduces transition probability spaces and transitional random variables.
result Satisfies all desired relevance relations except symmetry.
The existence of stationary Markov perfect equilibria in stochastic games is shown under a general condition called "(decomposable) coarser transition kernels". This result covers various earlier existence results on correlated equilibria, noisy stochastic games, stochastic games with finite actions and state-independe…
Improved KSD test for better detection of differences in distributions.
problem Low power of KSD test when distributions have same modes but different mixing proportions.
method Perturb the observed sample using Markov transition kernels to improve KSD test power.
result Perturbed KSD test can lead to substantially higher power than the original KSD test.
Stochastic kernel based dimensionality reduction approaches have become popular in the last decade. The central component of many of these methods is a symmetric kernel that quantifies the vicinity between pairs of data points and a kernel-induced Markov chain on the data. Typically, the Markov chain is fully specified…
New algorithm achieves data-dependent regret bounds in MDPs with unknown transitions.
problem Achieving best-of-both-worlds guarantees with data-dependent regret bounds in MDPs with unknown transitions.
method Optimistic follow-the-regularized-leader algorithm with new optimistic Q-function estimators and transition bonus.
result First-order, second-order, and path-length bounds with polylog(T) regret in the stochastic regime.
We develop algorithms with low regret for learning episodic Markov decision processes based on kernel approximation techniques. The algorithms are based on both the Upper Confidence Bound (UCB) as well as Posterior or Thompson Sampling (PSRL) philosophies, and work in the general setting of continuous state and action …
We consider online learning for minimizing regret in unknown, episodic Markov decision processes (MDPs) with continuous states and actions. We develop variants of the UCRL and posterior sampling algorithms that employ nonparametric Gaussian process priors to generalize across the state and action spaces. When the trans…
New method trains Markov kernels for efficient sampling.
problem Efficient sampling from complex probability distributions.
method Adversarial learning of involutive Metropolis-Hastings kernels.
result Minimizes total variation distance to empirical data.
We study optimal solutions to an abstract optimization problem for measures, which is a generalization of classical variational problems in information theory and statistical physics. In the classical problems, information and relative entropy are defined using the Kullback-Leibler divergence, and for this reason optim…
A multi-task GP model tracks time-varying transition probabilities between two states.
problem Tracking time-varying transition probabilities between 'moves' and 'pauses' states.
method Kernel-based multi-task Gaussian Process model with time-variability and constraints.
result Enforces constraints while learning transition probabilities.
Researchers approximate conditional expectation operators using kernel methods.
problem Statistical approximation of conditional expectation operators under minimal assumptions.
method Modifying the domain of the operator, approximating it by Hilbert-Schmidt operators in a reproducing kernel Hilbert space.
result The nonparametric estimate of the operator converges to a specific limiting object.
HDT improves MCMC on graphs with history-dependent sampling.
problem Efficient sampling from target distributions on general graphs with low computational overhead.
method History-driven target (HDT) framework that replaces the original target distribution with a history-dependent one.
result Near-zero variance performance and scalability to large graphs with memory-efficient implementation.
Paper tackles matrix estimation under arbitrary noise, achieving minimax optimality.
problem Noisy low-rank-plus-sparse matrix recovery under arbitrary dependence.
method Incoherent-constrained least-square estimator, novel energy spreading result.
result Achieves minimax optimality in estimating structured Markov transition kernels.
The paper studies convergence of kernel autocovariance operators for stationary processes.
problem Estimating autocovariance operators of stationary processes on Polish spaces.
method Investigates convergence of empirical estimates of autocovariance operators under various conditions.
result Provides consistency results for kernel PCA and spectral analysis methods.
Extended elliptical slice sampling for infinite-dimensional spaces, proving reversibility.
problem Proving reversibility of elliptical slice sampling in infinite-dimensional spaces.
method Extended elliptical slice sampling to infinite-dimensional separable Hilbert spaces, providing an alternative proof of reversibility.
result The approach yields a positive semi-definite Markov operator, proving reversibility.
DenseHMM improves HMMs by learning dense representations that enable gradient-based optimization.
problem Learning dense representations for hidden states and observables in HMMs.
method DenseHMM uses kernelized transition probabilities and two optimization schemes.
result DenseHMM achieves superior performance and expressiveness compared to standard HMMs.
Adapts flow matching for MCMC to improve sampling efficiency.
problem Improving sampling efficiency in MCMC for complex distributions.
method Combines Markov chain and CNFs to learn a path between distributions.
result Achieves similar performance to state-of-the-art methods but with lower computational cost.
New method improves RL in continuous spaces with kernel smoothing.
problem Sample efficiency and structural assumptions in classical RL.
method Kernel smoothing model-based approach with Bernstein-style exploration bonus.
result Achieves improved regret bound in finite-horizon settings.
Consider a reference Markov process with initial distribution π0 and transition kernels {Mt}t∈[1:T], for some T∈N. Assume that you are given distribution πT, which is not equal to the marginal distribution of the reference process at time T. In this scenario, Schrödinger addressed t…
Bootstrap method for Markov chains in reinforcement learning.
problem Distributional consistency in finite controlled Markov chains with unknown control policies.
method Model-based bootstrap with novel LLN and CLT for visitation counts and transition increments.
result Asymptotically valid confidence intervals for value and Q-functions in offline RL. This study analyzes convergence and stability of reinforcement learning algorithms.
problem Understanding the conditions under which reinforcement learning algorithms converge and remain stable.
method Theoretical analysis of convergence and stability of Episodic Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning, and Online Decision Transformers.
result The algorithms can achieve near-optimal behavior if the transition kernel is close to a deterministic kernel.
New neural processes use stacked Markov operators to improve flexibility.
problem Improving flexibility in neural processes.
method Stacking neural parameterized Markov transition operators in function space.
result MNPs outperform baseline models on various tasks.
Kernel-based methods improve policy evaluation in MRP models.
problem Estimating value functions in infinite-horizon discounted MRP models.
method Kernel-based temporal difference methods using reproducing kernel Hilbert spaces.
result Optimal error bounds derived for the kernel-based LSTD estimate.
We report an exact likelihood computation for Linear Gaussian Markov processes that is more scalable than existing algorithms for complex models and sparsely sampled signals. Better scaling is achieved through elimination of repeated computations in the Kalman likelihood, and by using the diagonalized form of the state…
Risk measures applied to dynamic Markov processes with varying risk aversion.
problem Investigating dynamic risk measures in Markov decision processes with varying risk aversion.
method Distributional viewpoint on law-invariant convex risk measures, applied to Markov decision processes with latent costs and random actions.
result Existence of optimal policies in finite and infinite time horizons under mild assumptions.
This paper proposes a stochastic model using the concept of Markov chains for the inter-state transitions of the millisecond order quasi-stable phase synchronized patterns or synchrostates, found in multi-channel Electroencephalogram (EEG) signals. First and second order transition probability matrices are estimated fo…
New insights into Markov chain geometry via positive transition measures.
problem Lack of statistical meaning in the space of transition probabilities.
method Constructing an extension of the space of transition probabilities using Amari's theory of positive measures.
result Introduction of a new dually flat structure for the space of positive transition measures.
Develops a flexible model for regime transitions in time series data.
problem Nonlinear and context-dependent regime transitions in time series data.
method Semi-parametric state-space model with learned transition functions.
result Improved recovery of nonlinear transition dynamics and earlier detection of regime changes.
A new method learns robust policies from offline data with latent structures.
problem Conservative policies under unrealistic dynamics shifts.
method d-RRMDP framework with f-divergence regularization and R2PVI algorithm. result R2PVI learns robust policies with superior computational efficiency.
Develops CLTs for Markov chain transition probabilities and policies.
problem Estimating transition probabilities and policies in controlled Markov chains.
method Non-parametric estimator for transition matrices; CLTs for value, Q-, and advantage functions; goodness-of-fit tests.
result Asymptotic normality of estimators under specific logging policies.
New algorithm clusters trajectories from multiple Markov chains with near-optimal error.
problem Clustering trajectories from multiple unknown Markov chains.
method Two-stage algorithm: spectral clustering followed by likelihood-based refinement.
result Achieves near-optimal clustering error with high probability.
Neural models learn continuous-time Markov chain transition rates from data.
problem Learning transition rates for complex stochastic systems.
method Neural networks to model nonlinear transition rates from observed data.
result Neural models outperform traditional methods in accuracy.
Paper learns meaningful state and action representations from MDP trajectories.
problem Learning good state and action representations from MDP trajectories.
method Tensor decomposition, kernelization, importance sampling, low-Tucker-rank approximation.
result The learned state/action abstractions provide accurate approximations to latent block structures.
Model detects market anomalies using a Hawkes process with hidden Markov chain.
problem Detecting high-frequency market manipulation in cryptocurrency trades.
method Developed a Markov-modulated Hawkes process with piecewise constant excitation kernels.
result Demonstrated the model's effectiveness in detecting suspicious trading activities.
New method links covariates to CTMCs using RKHS, improving state transitions modeling.
problem Traditional multistate models rely on linear relationships, limiting flexibility.
method Nonparametric approach using RKHS, with Frequentist and Bayesian versions.
result Effective in identifying nonlinear transition functions and predicting long-term behaviors.
This paper studies how to find compact state embeddings from high-dimensional Markov state trajectories, where the transition kernel has a small intrinsic rank. In the spirit of diffusion map, we propose an efficient method for learning a low-dimensional state embedding and capturing the process's dynamics. This idea a…
A Markov Chain approach for aligning generative models from pairwise human preferences.
problem Aligning generative models from pairwise human preferences.
method Markov Chain from Human Feedback (MCHF)
result MCHF converges geometrically fast to the stationary distribution.
New algorithm detects changes in Markov kernels with unknown post-change kernel.
problem Detecting changes in Markov kernels with unknown post-change kernel.
method Developed a new change detection algorithm assuming uniform ergodicity.
result Derived upper and lower bounds on mean delay and time between false alarms.
Deeptime simplifies learning dynamical models from time series data.
problem Understanding complex systems through dynamical models from time series data.
method Various tools for estimating dynamical models including conventional and kernel/deep learning methods.
result Estimates dynamical models from time series data efficiently and with rich analysis methods.
Scalable Bayesian sampling is playing an important role in modern machine learning, especially in the fast-developed unsupervised-(deep)-learning models. While tremendous progresses have been achieved via scalable Bayesian sampling such as stochastic gradient MCMC (SG-MCMC) and Stein variational gradient descent (SVGD)…
New algorithms reduce regret in reinforcement learning with MNL approximations.
problem Efficient reinforcement learning with MNL function approximation for MDPs.
method Proposed randomized exploration algorithms with frequentist regret guarantees.
result Achieved improved regret bounds for MNL transition models.
We develop efficient and sharp bounds on policy value under perturbations in MDPs.
problem Evaluating policies under best- and worst-case perturbations in MDPs with transition observations.
method Proposed a perturbation model for MDPs, developed semiparametrically efficient estimator with asymptotic normality.
result Semiparametrically efficient and asymptotically normal estimator for policy value bounds.
In most sampling algorithms, including Hamiltonian Monte Carlo, transition rates between states correspond to the probability of making a transition in a single time step, and are constrained to be less than or equal to 1. We derive a Hamiltonian Monte Carlo algorithm using a continuous time Markov jump process, and ar…
Improved algorithm detects changes in RL environments with non-stationary MDPs.
problem Learning in non-stationary reinforcement learning environments.
method R-BOCPD-UCRL2 algorithm for MDPs with multinomial state transitions.
result Near-optimal theoretical guarantees in terms of false-alarm rate and detection delay.
TMTF improves time series visualization by separating dynamic regimes.
problem Misleading global transition matrix in time series analysis.
method Temporal chunking, local transition matrices, and image assembly.
result Temporal segmentation reveals distinct transition dynamics.
Proposes MIVI for efficient posterior estimation and design of MCMC transitions.
problem Efficiently estimating posterior distributions in constrained time.
method Combines variational inference and MCMC with a variational distribution and optimized Markov chain.
result Optimized Markov chain improves variational distribution and vice versa, leading to more accurate posteriors.