This paper first describes a class of uncertain stochastic control systems with Markovian switching, and derives an Itô-Liu formula for Markov-modulated processes. And we characterize an optimal control law, which satisfies the generalized Hamilton-Jacobi-Bellman (HJB) equation with Markovian switching. Then, by using …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “Markovian policies”
New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
Optimizes state monitoring in Markovian systems with cost constraints.
problem Balancing state queries with prediction costs in Markovian systems.
method Greedy policy and SGD-based learning variant for optimal predict-query tradeoff.
result Greedy policy is suboptimal but performs close to optimal under certain conditions.
New estimator reduces bias-variance tradeoff in Markovian interference experiments.
problem Estimating impact of interventions in systems with limited resources.
method Differences-In-Q (DQ) estimator for on-policy policy evaluation.
result DQ estimator has exponentially smaller variance than off-policy methods.
This work expands state-action aggregation methods for non-Markovian environments.
problem Real-world problems with large state and action spaces are not tractable with existing methods.
method Expands Extreme State Aggregation (ESA) framework to non-Markovian homomorphisms and relaxes policy uniformity.
result Near-optimal performance is guaranteed even for non-Markovian homomorphisms.
We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account…
Stochastic differential equation approximation for linear TD(0) under Markovian noise
problem Temporal-difference learning with linear function approximation
method Stochastic differential equation approximation
result Explains the constant-stepsize error floor
Paper analyzes Greedy-GQ for reinforcement learning with Markovian noise.
problem Analyzing Greedy-GQ for reinforcement learning with Markovian noise.
method Develops finite-sample analysis for Greedy-GQ with linear function approximation under Markovian noise.
result Provides theoretical justification for choosing stepsizes for faster convergence.
Paper establishes convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
problem Analyzing convergence rates and concentration bounds for stochastic approximation and reinforcement learning with Markovian noise.
method Novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing length.
result First almost sure convergence rate and maximal concentration bound with exponential tails for contractive stochastic approximation algorithms with Markovian noise.
This paper extends policy gradient methods to partially observable environments.
problem Learning optimal policies in partially observable environments.
method Developed new tools including advantage function to generalize policy gradient algorithms and study their convergence in partially observable Markovian policies.
result Generalized theoretical guarantees of policy gradient algorithms to partially observable domains.
Paper tackles robust offline RL for non-Markovian processes, improving efficiency and applicability.
problem Learning robust policies for non-Markovian decision processes with limited offline data.
method Proposes a novel algorithm with dataset distillation and LCB design for robust values, derived new dual forms, and introduces concentrability coefficients.
result Proves polynomial sample efficiency for finding ε-optimal robust policies.
Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.
problem Lack of structured representation for non-Markovian stochastic rewards in reinforcement learning.
method Introduces probabilistic reward machines (PRMs) and presents an algorithm to learn them from decision processes.
result Algorithm proves correct and convergent for learning PRMs from decision processes.
Paper analyzes finite-time performance of SA in RL with Markovian noise.
problem Finite-time analysis of linear two-timescale stochastic approximation with Markovian noise.
method Finite-time analysis of linear two-timescale SA with Markovian noise, considering both transient and steady-state terms.
result No discrepancy in convergence rate between Markovian and martingale noise; transient term is and steady-state term is .
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Samplingcs.LG
Paper analyzes convergence of Adam-type RL algorithms under Markovian sampling.
problem Theoretical convergence analysis of Adam-type RL algorithms.
method Develops techniques for analyzing convergence under Markovian sampling.
result PG-AMSGrad and TD-AMSGrad converge to stationary points or global optima at specified rates.
Describes state variables in sequential decision problems, linking them to Markovian and non-Markovian models.
problem Sequential decision problems, especially in active learning and POMDPs, where decisions affect what is observed and learned.
method Canonical framework and novel two-agent perspective of POMDPs, defining state variables to claim Markovian or non-Markovian models.
result Properly modeled sequential decision problems are Markovian, while real decision problems are often non-Markovian.
New algorithm prices Bermudan options using Wiener chaos expansion for non-Markovian processes.
problem Pricing Bermudan options with non-Markovian payoff processes.
method Modified Longstaff Schwartz algorithm with Wiener chaos expansion for non-Markovian settings.
result Embarrassingly parallel algorithm for efficient computation.
DeepSynth synthesizes automata to guide deep RL agents through sparse, non-Markovian rewards.
problem Training deep RL agents with sparse, non-Markovian rewards and unknown high-level objectives.
method Employing a novel algorithm for synthesizing compact automata to uncover sequential structure from trace data.
result Reduces the number of iterations required for policy synthesis by two orders of magnitude and improves scalability.
Partially observable environments present an important open challenge in the domain of sequential control learning with delayed rewards. Despite numerous attempts during the two last decades, the majority of reinforcement learning algorithms and associated approximate models, applied to this context, still assume Marko…
Improved TD learning for non-i.i.d. Markovian data.
problem Convergence analysis of two time-scale TD learning under Markovian samples.
method Non-asymptotic convergence analysis of two time-scale TD with gradient correction under Markovian data.
result Two time-scale TD can converge as fast as O(log t/(t^(2/3))) under diminishing stepsize.
This study improves convergence of two-timescale SA under Markovian noise in reinforcement learning.
problem Stability and convergence of two-timescale stochastic approximations under Markovian noise.
method Introduced a new control strategy for the fast timescale parameter.
result Established almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
Paper tackles efficient off-policy evaluation in long-horizon settings.
problem Efficient off-policy evaluation in long-horizon settings with diminishing overlap.
method Derives efficiency bounds for OPE under Markovian and time-invariant structures, develops a new DRL estimator.
result DRL estimator provides efficient OPE even with just one dependent trajectory in time-invariant Markov decision processes.
Paper compares UCB policy to new adaptive RL methods.
problem Optimal adaptive policies for Markovian decision processes with unknown transition probabilities.
method Compared UCB policy with MDP-Deterministic Minimum Empirical Divergence and Posterior sampling methods.
result MDP-DMED outperforms UCB in the tested RL scenarios.
FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Financeq-fin.CP
FinFlowRL learns from experts to optimize financial control in changing markets.
problem Traditional finance control methods fail in real-world, non-stationary markets.
method Imitation-Reinforcement Learning framework that pretrains on expert strategies and finetunes in noise space.
result Consistently outperforms individually optimized experts across diverse market conditions.
This paper improves sample complexity for AC and NAC algorithms under Markovian sampling.
problem Improving sample complexity for actor-critic and natural actor-critic algorithms.
method Characterizes convergence rate and sample complexity under Markovian sampling and mini-batch data.
result Improves sample complexity for AC and NAC algorithms by orders of magnitude.
Minimal assumptions analysis of Q-learning with time-varying policies.
problem Finite-time analysis of Q-learning with time-varying policies for discounted MDPs.
method Minimal assumptions, Poisson equation decomposition, sensitivity analysis.
result Established convergence rate and sample complexity for Q-learning.
Study designs incentives for adapting multi-agent systems without knowing their learning dynamics.
problem Designing incentives for an adapting population in multi-agent systems without prior knowledge of their learning dynamics.
method Introduces a model-based non-episodic Reinforcement Learning (RL) formulation for steering Markovian agents towards desired policies, focusing on history-dependent strategies to handle model uncertainty.
result Identifies conditions for the existence of steering strategies to guide agents to desired policies and provides empirical algorithms to approximately solve the objective.
Breaks down complex nonlinear dynamics into simpler components.
problem Control of nonlinear dynamical systems remains challenging.
method Inspired by hybrid switching systems, decomposes dynamics into simpler stochastic switching linear dynamical systems.
result Extracts hierarchies of Markovian and auto-regressive locally linear controllers from nonlinear experts.
Self-supervised approach improves reinforcement learning with attention.
problem Previous attempts at integrating attention with reinforcement learning failed to produce significant improvements.
method Utilizes Markovian properties of state input and multiple simultaneous foci of attention.
result New state-of-the-art results in the Arcade Learning Environment.
In this paper, we consider the optimal dividend problem for a company. We describe the surplus process of the company by a diffusion model with regime switching. The aim of the company is to choose a dividend policy to maximize the expected total discounted payments until ruin. In this article, we consider a hybrid div…
Paper analyzes convergence rates of two time-scale AC and NAC algorithms.
problem Finite-sample convergence rate analysis of two time-scale AC and NAC algorithms.
method Developed novel techniques for bias error and convergence rate analysis.
result Established non-asymptotic convergence rates for two time-scale AC and NAC.
MER algorithm speeds up VI solving with Markovian data.
problem Solving stochastic variational inequalities with Markovian data.
method MER algorithm using multi-scale sampling from a Markovian buffer.
result Achieves faster convergence without knowing Markov chain mixing time.
Consider the problem of a government that wants to reduce the debt-to-GDP (gross domestic product) ratio of a country. The government aims at choosing a debt reduction policy which minimises the total expected cost of having debt, plus the total expected cost of interventions on the debt ratio. We model this problem as…
Paper introduces IO-NPF for efficient Bayesian experimental design.
problem Efficient Bayesian experimental design in non-exchangeable settings.
method Inside-Out Nested Particle Filter (IO-NPF) for non-Markovian state-space models.
result IO-NPF achieves computational complexity, improving efficiency.
Improved TD learning reduces variance and bias errors.
problem Inefficient optimization variance in TD learning.
method Proposed a mathematically solid analysis of VRTD, showing linear convergence rate and reduced variance and bias errors.
result VRTD converges to a fixed-point solution with reduced variance and bias errors compared to vanilla TD.
FPG uses fractional calculus for efficient reinforcement learning with long-term memory.
problem High variance and inefficient sampling in standard policy gradient methods for long-term temporal modeling.
method Fractional Policy Gradients (FPG) incorporating Caputo fractional derivatives for power-law temporal correlations.
result Achieves asymptotic variance reduction of order O(t^(-alpha)) and sample efficiency gains.
New method optimizes decision-making in uncertain environments.
problem Optimal decision-making under partial observability.
method Nested sequential Monte Carlo algorithm for continuous POMDPs.
result Demonstrated effectiveness on continuous POMDP benchmarks.
New RL algorithms correct bias in dynamic data analysis.
problem Dynamic data generation and analysis create endogeneity issues.
method Instrument variable (IV)-based reinforcement learning (RL) algorithms.
result Established theoretical properties of IV-RL algorithms.
Bayesian RL enhances LLMs to reflectively explore and correct errors.
problem LLMs trained via RL lack reflective behaviors like rethinking and error correction.
method Bayesian RL framework that optimizes expected return under posterior distribution over Markov decision processes.
result BARL algorithm improves LLM performance in reasoning tasks.
A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscs.LG
Unified framework for finite-sample RL algorithms using Lyapunov theory.
problem Finite-sample convergence guarantees of asynchronous RL algorithms.
method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.
result Mean-square error bounds and convergence for various RL algorithms.
This paper analyzes the sample complexity of two timescale reinforcement learning algorithms.
problem Analyzing the sample complexity of two timescale reinforcement learning algorithms.
method Non-asymptotic analysis of linear and nonlinear TDC and Greedy-GQ algorithms under Markovian sampling with constant stepsize.
result The paper provides non-asymptotic convergence results for two timescale linear and nonlinear TDC and Greedy-GQ algorithms.
The paper explains why estimating a history-dependent policy can reduce MSE in reinforcement learning.
problem Understanding why history-dependent policies can improve MSE in off-policy evaluation.
method The paper derives a bias-variance decomposition of MSE for various OPE estimators, showing how history-dependent policies can decrease variance and increase bias.
result History-dependent policies can decrease the variance of importance sampling estimators, leading to lower MSE.
In this paper we propose a novel gradient algorithm to learn a policy from an expert's observed behavior assuming that the expert behaves optimally with respect to some unknown reward function of a Markovian Decision Problem. The algorithm's aim is to find a reward function such that the resulting optimal policy matche…
This paper improves convergence bounds for AC and NAC algorithms with function approximation.
problem Improving convergence bounds for actor-critic algorithms with function approximation.
method Non-asymptotic analysis of AC and NAC algorithms with compatible function approximation.
result Eliminates the term ε_critic from the error bounds while maintaining best known sample complexities.
The paper studies risk-sensitive MDPs with recursive risk measures.
problem Risk-sensitive decision-making in MDPs with unbounded costs.
method Recursive application of static risk measures, Bellman equation derivation, existence of optimal policies.
result Existence of Markovian optimal policies for infinite planning horizons, contractive model for stationary optimal policy.
Develops a method to learn optimal timing of treatments from observational data.
problem Choosing the right time to start treatments in dynamic decision-making problems.
method Advantage Doubly Robust Estimator for dynamic treatment rules under sequential ignorability.
result Proves welfare regret bounds and shows promising empirical performance.
Anticipatory Reinforcement Learning: From Generative Path-Laws to Distributional Value Functionscs.LG
ARL bridges non-Markovian decision processes with reinforcement learning, improving foresight and stability.
problem Inaccurate foresight in non-Markovian environments due to state-based methods' limitations.
method Lifted state space into a signature-augmented manifold, using a self-consistent field approach to anticipate future path-law.
result ARL achieves deterministic evaluation of expected returns with reduced computational complexity and variance.
Resemblance of the power-law scaling behavior of a non-Markovian and nonlinear point processescond-mat.stat-mech
Non-Markovian point process shows power-law scaling, similar to nonlinear Markovian process.
problem Understanding the scaling behavior of non-Markovian point processes.
method Analyzed a confined fractional Brownian motion-driven point process and compared it to a nonlinear Markovian process.
result A nonlinear Markovian process can reproduce the power-law scaling behavior of a non-Markovian point process.