Describes state variables in sequential decision problems, linking them to Markovian and non-Markovian models.
problem Sequential decision problems, especially in active learning and POMDPs, where decisions affect what is observed and learned.
method Canonical framework and novel two-agent perspective of POMDPs, defining state variables to claim Markovian or non-Markovian models.
result Properly modeled sequential decision problems are Markovian, while real decision problems are often non-Markovian.
Study improves covariance estimation for SGD under Markovian data, matching best rates.
problem Improving covariance estimation for SGD in Markovian data settings.
method Online overlapping batch-means covariance estimator for SGD under Markovian sampling.
result Established convergence rates for covariance estimation under Markovian sampling.
The paper develops a Hoeffding inequality for Markov chains and applies it to bandit problems.
problem Developing a Hoeffding inequality for Markov chains and applying it to bandit problems.
method Developed a Hoeffding inequality for the partial sums of an irreducible Markov chain on a finite state space.
result Demonstrated the inequality's effectiveness in identifying approximately best Markovian arms and minimizing regret in Markovian bandits.
State spaces of multifactor approximations of nonnegative Volterra processes are linear transformations of the nonnegative orthant.
problem Characterizing state spaces of multifactor approximations of nonnegative Volterra processes.
method Explicit linear transformation of the nonnegative orthant.
result State spaces of multifactor approximations of nonnegative Volterra processes are given by explicit linear transformation of the nonnegative orthant.
Optimizes state monitoring in Markovian systems with cost constraints.
problem Balancing state queries with prediction costs in Markovian systems.
method Greedy policy and SGD-based learning variant for optimal predict-query tradeoff.
result Greedy policy is suboptimal but performs close to optimal under certain conditions.
Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.
problem Lack of structured representation for non-Markovian stochastic rewards in reinforcement learning.
method Introduces probabilistic reward machines (PRMs) and presents an algorithm to learn them from decision processes.
result Algorithm proves correct and convergent for learning PRMs from decision processes.
Modeling high-frequency order book data with Hawkes-Markovian process.
problem Capturing the dynamics of high-frequency order book events.
method Hawkes process with Markovian baseline intensities, LASSO regularization, and Akaike Information Criteria.
result Effective modeling of order book dynamics with reduced parameter redundancy.
The paper analyzes multivariate payments in multi-state life insurance using Markovian state processes.
problem Analyzing joint effects of life annuities and death benefits in a multi-state framework.
method Introduces multivariate present value of future payments, derives differential equations and moment generating functions, and focuses on pair-wise covariances.
result Derives Hattendorff type results for pair-wise covariances in a disability model.
Most real-world problems have huge state and/or action spaces. Therefore, a naive application of existing tabular solution methods is not tractable on such problems. Nonetheless, these solution methods are quite useful if an agent has access to a relatively small state-action space homomorphism of the true environment …
HS-FNO models non-Markovian PDEs by learning history and future states.
problem Non-Markovian dynamics where future states depend on past history.
method History-Space Fourier Neural Operator (HS-FNO) for delay and memory-driven PDEs.
result HS-FNO achieves lowest aggregate errors across various PDE families.
The purpose of this article is to describe all possible beliefs of market participants on objective measures under Markovian environments when a risk-neutral measure is given. To achieve this, we employ the Martin integral representation of Markovian pricing kernels. Then, we offer economic and financial implications o…
We study (backward) stochastic differential equations with noise coming from a finite state Markov chain. We show that, for the solutions of these equations to be `Markovian', in the sense that they are deterministic functions of the state of the underlying chain, the integrand must be of a specific form. This allows u…
This work tackles large action spaces in RL by binarizing actions.
problem Large action spaces in reinforcement learning cause significant challenges.
method Sequentializing actions and binarizing the action space.
result Binarizing the action space can significantly improve RL algorithms and reduce state space size.
The paper develops a deep signature approach for option pricing under non-Markovian stochastic volatility models.
problem Pricing options under non-Markovian stochastic volatility models is challenging due to the dependence on historical paths.
method Reformulate the asset dynamics as a rough stochastic differential equation and represent rough paths via signatures. Apply standard analytical tools to solve the transformed equation.
result The deep signature approach provides a theoretically grounded and computationally efficient framework for option pricing.
Develops RL algorithm for non-Markovian, non-stationary reward streams.
problem Maximizing rewards from non-Markovian, non-stationary reward streams.
method Uses causal DAG to construct Markov states, solves periodic MDP.
result Optimal state construction maximizes discounted rewards.
In this paper we present a model for the hidden Markovian bandit problem with linear rewards. As opposed to current work on Markovian bandits, we do not assume that the state is known to the decision maker before making the decision. Furthermore, we assume structural side information where the decision maker knows in a…
We establish existence, uniqueness and regularity of solution results for a class of backward stochastic partial differential equations with singular terminal condition. The equation describes the value function of non-Markovian stochastic optimal control problem in which the terminal state of the controlled process is…
This paper develops a spectral theory of Markovian asset pricing models where the underlying economic uncertainty follows a continuous-time Markov process X with a general state space (Borel right process (BRP)) and the stochastic discount factor (SDF) is a positive semimartingale multiplicative functional of X. A key …
We describe a Markov latent state space (MLSS) model, where the latent state distribution is a decaying mixture over multiple past states. We present a simple sampling algorithm that allows to approximate such high-order MLSS with fixed time and memory costs.
New estimator reduces bias-variance tradeoff in Markovian interference experiments.
problem Estimating impact of interventions in systems with limited resources.
method Differences-In-Q (DQ) estimator for on-policy policy evaluation.
result DQ estimator has exponentially smaller variance than off-policy methods.
A new method for online VI in SSMs using asymptotic contrast.
problem Lack of functionality for streaming data in standard VI methods for SSMs.
method Propose maximising an IWAE-type variational lower bound on the asymptotic contrast function using stochastic approximation.
result OSIWAE allows for online learning of model parameters and latent states.
Markovian RNN adapts to nonstationary data using HMM for better time series prediction.
problem Nonstationary sequential data in real-life applications.
method Markovian RNN with HMM for regime switching and end-to-end optimization.
result Significant performance gains over vanilla RNN and Markov Switching ARIMA.
Estimates hybrid dynamical systems with polynomial expansions and Markovian switching.
problem Identifying hybrid dynamical systems with nonlinear autoregressive exogenous (NARX) components and Markovian switching.
method Probabilistic framework using Expectation Maximization for parameter estimation, including submodel coefficients, hidden state values, and transition probabilities. Disentangles mode classification and NARX regression tasks. Uses soft-labels and coordinate descent approach for parameter fitting.
result Demonstrated on a SMNARX problem with three nonlinear sub-models, achieving parsimonious models through l1-norm bridge estimation and hard-thresholding.
Paper analyzes finite-time performance of SA in RL with Markovian noise.
problem Finite-time analysis of linear two-timescale stochastic approximation with Markovian noise.
method Finite-time analysis of linear two-timescale SA with Markovian noise, considering both transient and steady-state terms.
result No discrepancy in convergence rate between Markovian and martingale noise; transient term is o(1/kc) and steady-state term is O(1/k). Estimates dependent parameters using Markovian dependence with shrinkage.
problem Estimating dependent parameters from a hidden Markov model.
method Developed a novel non-parametric shrinkage algorithm combining Tweedie-based ideas and efficient state estimation.
result Superior performance compared to non-shrinkage methods in hidden Markov models.
PMP extends GNNs to handle past states efficiently.
problem Efficient querying of data structures dependent on previous states.
method Persistent Message Passing (PMP) which persists past states through new nodes.
result Significantly outperforms GNNs in handling out-of-distribution data.
New approach tackles non-Markovian behavior in maternal health programs.
problem Improving adherence and engagement in maternal and child healthcare programs.
method Extending RMABs to non-Markovian settings, using time-series forecasting and TARI policy.
result Significant increase in engagement and content listened compared to existing methods.
New RL algorithms learn Gittins indices for unknown Markovian states.
problem Learning Gittins indices for unknown Markovian state transitions.
method Tabular (QGI) and Deep RL (DGN) algorithms based on retirement formulation.
result Lower run time, less storage space, better convergence to Gittins index.
Non-Markovian point process shows power-law scaling, similar to nonlinear Markovian process.
problem Understanding the scaling behavior of non-Markovian point processes.
method Analyzed a confined fractional Brownian motion-driven point process and compared it to a nonlinear Markovian process.
result A nonlinear Markovian process can reproduce the power-law scaling behavior of a non-Markovian point process.
Paper introduces IO-NPF for efficient Bayesian experimental design.
problem Efficient Bayesian experimental design in non-exchangeable settings.
method Inside-Out Nested Particle Filter (IO-NPF) for non-Markovian state-space models.
result IO-NPF achieves O(T2) computational complexity, improving efficiency. A new method scales Gaussian process variational autoencoders to handle high-dimensional time series.
problem Scalability issue in Gaussian process variational autoencoders (GPVAEs).
method Introducing Markovian GPs and using Kalman filtering and smoothing for linear time training.
result MGPVAE outperforms existing approaches in various tasks with high scalability.
A new method predicts non-Markovian closure terms for complex systems.
problem Predicting the effect of unresolved variables on resolved dynamics in high-dimensional systems.
method Mamba-Assisted Closure (MAC) framework: sequence model trained to predict closure from resolved trajectory, coupled with reduced-order equations.
result Substantially outperforms existing methods in predictive accuracy and long-time stability.
Projects Markovian processes from Itô semimartingales with jumps.
problem Modeling Itô semimartingales with jumps using Markovian projections.
method Construct Markovian projections for Itô semimartingales with jumps using non-local FPKEs.
result Markovian projections match the marginal laws of the original process.
A method for optimizing under unknown Markovian data distributions.
problem Optimizing under unknown and indirectly observed probability distributions via Markovian data.
method Data-driven distributionally robust optimization model with Frank-Wolfe algorithm.
result The proposed method finds a stationary point efficiently and outperforms state-of-the-art methods.
We revisit the development of grid based recursive approximate filtering of general Markov processes in discrete time, partially observed in conditionally Gaussian noise. The grid based filters considered rely on two types of state quantization: The \textit{Markovian} type and the \textit{marginal} type. We propose a s…
A new metric detects non-Markovian states in partially observable environments.
problem Learning state representations in partially observable environments.
method Introducing the λ-discrepancy metric to detect non-Markovian states. result The λ-discrepancy is zero for Markov processes and non-zero for partially observable environments. Study efficient algorithms for nonconvex optimization with state-dependent Markov data.
problem Stochastic optimization with Markovian data and state-dependent transition kernels.
method Projection-based and projection-free algorithms for constrained nonconvex problems.
result The number of oracle calls to achieve an ε-stationary point is O(1/ε2.5). Enhances reward specification in RL with a novel language-based approach.
problem Reward specification in RL can lead to unintended, potentially harmful behaviours.
method Developed a novel class of language-based Reward Machines using RML's built-in memory.
result Can specify non-regular, non-Markovian reward functions for complex tasks.
This paper extends Markovian projections to semimartingales with jumps.
problem Extending Markovian projections to semimartingales with jumps.
method Using Markovian projections to match marginal laws of Itô semimartingales with jumps.
result Existence of Markovian projections for Itô semimartingales with jumps.
Partially observable environments present an important open challenge in the domain of sequential control learning with delayed rewards. Despite numerous attempts during the two last decades, the majority of reinforcement learning algorithms and associated approximate models, applied to this context, still assume Marko…
This paper solves the inversion problem for jump processes using Markovian projections.
problem Calibrating jump-diffusion models with both local and stochastic features.
method Inverting Markovian projections for pure jump processes.
result Constructs calibrated local stochastic intensity (LSI) models for credit risk applications.
A new framework models multi-state events and biomarkers.
problem Limited representation of complex multi-state trajectories.
method General multi-state joint modeling framework.
result Accurate parameter recovery and personalized predictions.
Paper tackles non-Markovian control problems with new learning methods.
problem Non-Markovian stochastic control problems with unknown parameters.
method Off-model training and importance sampling for deep neural network approximation.
result Quantitative error bounds for adaptive learning under model uncertainty.
The paper analyzes stability of random matrix products with Markovian noise.
problem Analyzing stability of random matrix products with Markovian noise.
method Using a super-Lyapunov drift condition and controlled growth of matrix-valued functions, the paper provides an exponential stability result for the p-th moment of random matrix product.
result Finite-time p-th moment bounds for linear stochastic approximation and TD learning algorithms.
SRMC framework reduces Monte Carlo variance by history-based sampling in high-dimensional spaces.
problem Efficient sampling in high-dimensional discrete or continuous state spaces.
method Score-Repellent Monte Carlo (SRMC) framework that summarizes history through running average of score evaluations.
result Improves estimator variance and mode coverage with constant memory usage.
Analyzes non-Markovian environments in stochastic approximation.
problem Understanding learning mechanisms in non-ergodic, non-Markovian settings.
method Analytic framework for transformer learning and continual learning.
result Proposes a new approach to transformer and continual learning.
ARL bridges non-Markovian decision processes with reinforcement learning, improving foresight and stability.
problem Inaccurate foresight in non-Markovian environments due to state-based methods' limitations.
method Lifted state space into a signature-augmented manifold, using a self-consistent field approach to anticipate future path-law.
result ARL achieves deterministic evaluation of expected returns with reduced computational complexity and variance.
A novel method reduces dimensionality for filtering SRNs with observed variables.
problem Challenges in estimating hidden state variables in SRNs with limited observations.
method Filtered Markovian Projection (Filtered MP) for dimensionality reduction in filtering.
result Filtered MP guarantees consistency and superior computational efficiency in high dimensions.