On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Factored Markov Decision Processes”

Towards Minimax Optimal Reinforcement Learning in Factored Markov Decision Processescs.LG

Study minimax optimal RL in factored MDPs with bonus exploration.

problem Optimal reinforcement learning in episodic factored MDPs.

method Proposes two model-based algorithms with bonus exploration for minimax optimal regret.

result Achieves minimax optimal regret guarantees for rich factored structures.

Reinforcement Learning Factored Markov Decision Processes Bonus Exploration Minimax Optimal

2020-06-24abs ↗pdf ↗

Finding Safe Zones of policies Markov Decision Processescs.LG

Algorithm finds safe zones in policy Markov Decision Processes to limit trajectory escape.

problem Finding safe zones in policy Markov Decision Processes to limit trajectory escape.

method Bi-criteria approximation learning algorithm with polynomial sample complexity.

result Achieves almost 2 approximation for both escape probability and safe zone size.

Markov Decision Processes Safe Zones Trajectory Escape Approximation Algorithms Polynomial Sample Complexity

2022-02-23abs ↗pdf ↗

Pricing bonds with optional sinking feature using Markov Decision Processesq-fin.PR

An efficient method to price bonds with optional sinking feature is presented. Such instruments equip their issuer with the option (but not the obligation) to redeem parts of the notional prior to maturity, therefore the future cash flows are random. In a one-factor model for the issuer's default intensity we show that…

2013-05-22abs ↗pdf ↗

Robust Batch Policy Learning in Markov Decision Processesmath.ST

The paper tackles robust policy learning in MDPs using statistical methods.

problem Offline data-driven sequential decision making in MDPs.

method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.

result Established a rate-optimal regret bound up to a logarithmic factor.

Markov Decision Processes Offline Policy Learning Statistical Efficiency Robust Policy Semi-parametric Statistics

2020-11-09abs ↗pdf ↗

Achieving

\varepsilon^{-2}

Dependence for Average-Reward Q-Learning with a New Contraction Principlecs.LG

New Q-learning method achieves optimal sample complexity for average-reward problems.

problem Challenges in achieving optimal sample complexity for average-reward Q-learning.

method Synchronous and asynchronous Q-learning with a new contraction principle.

result Optimal

\widetilde{O}(\varepsilon^{-2})

sample complexity guarantees.

Q-learning Markov decision processes Contraction principle Average-reward Sample complexity

2026-01-29abs ↗pdf ↗

Is Temporal Difference Learning Optimal? An Instance-Dependent Analysisstat.ML

New methods improve temporal difference learning for policy evaluation in Markov decision processes.

problem Improving temporal difference learning for policy evaluation in Markov decision processes.

method Introduced variance-reduced forms of stochastic approximation to achieve non-asymptotic, instance-dependent optimality.

result Temporal difference learning is strictly suboptimal, but variance-reduced forms achieve optimality up to logarithmic factors.

temporal difference learning policy evaluation Markov decision processes stochastic approximation variance reduction

2020-03-16abs ↗pdf ↗

Multi-agent Reinforcement Learning for Networked System Controlcs.LG

This paper improves MARL for networked systems through new protocols and discount factors.

problem Improving control in networked systems using multi-agent reinforcement learning.

method Formulated as a spatiotemporal Markov decision process, introduced a spatial discount factor, and proposed NeurComm.

result Appropriate spatial discount factor enhances learning curves of non-communicative MARL algorithms.

multi-agent reinforcement learning networked systems spatiotemporal Markov decision process spatial discount factor neurcomm

2020-04-03abs ↗pdf ↗

Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximationcs.LG

A new method for risk-averse decision-making in Markov processes with improved regret bounds.

problem Risk-averse decision-making in Markov processes.

method Introduces mini-batch measures and multipattern risk-averse problems in a feature-based

Q

-learning method.

result Proves a high-probability regret bound of

\mathcal{O}\big(H^2 N^H \sqrt{ K}\big)

for the

Q

-learning method.

Reinforcement Learning Markov Decision Processes Risk Measures Mini-batch Measures Multipattern Risk

2026-05-01abs ↗pdf ↗

Alpha Discovery via Grammar-Guided Learning and Searchq-fin.CP

AlphaCFG discovers alpha factors using grammar-guided search.

problem Discovering formulaic alpha factors in finance.

method AlphaCFG uses a grammar-based framework to define and discover alpha factors with syntactic and semantic constraints.

result AlphaCFG outperforms state-of-the-art methods in trading profitability and efficiency.

quantitative finance alpha factors grammar-guided search Monte Carlo Tree Search financial interpretation

2026-01-29abs ↗pdf ↗

Style Miner: Find Significant and Stable Explanatory Factors in Time Series with Constrained Reinforcement Learningcs.LG

Style Miner generates stable and significant style factors for time series analysis.

problem Finding significant and stable explanatory factors in high-dimensional time series data.

method Proposes a reinforcement learning method to balance explanatory power and stability constraints.

result Outperforms existing methods by a large margin and achieves a 10% gain in R-squared explanatory power.

reinforcement learning time series analysis style factors explanatory power stability

2023-03-21abs ↗pdf ↗

Causal Markov Decision Processes: Learning Good Interventions Efficientlystat.ML

New formalism for decision making combines causal structures with MDPs, improving reinforcement learning performance.

problem Sequential decision making with causal knowledge to improve performance.

method Causal Markov Decision Processes (C-MDPs) and C-UCBVI algorithm exploiting causal structure.

result C-UCBVI achieves an

ilde{O}(HS\sqrt{ZT})

regret bound, independent of actions.

Causal Markov Decision Processes Reinforcement Learning Sequential Decision Making Causal Structures UCBVI

2021-02-15abs ↗pdf ↗

EB-dynaRE: Real-Time Adjustor for Brownian Movement with Examples of Predicting Stock Trends Based on a Novel Event-Based Supervised Learning Algorithmq-fin.GN

Model predicts stock prices using GAN and RoI Pooling.

problem Predicting stock prices influenced by macroeconomic factors.

method Markov Decision Process, GAN, RoI Pooling.

result Identifies macroeconomic factors' influence on stock prices.

stock prediction GAN RoI Pooling Markov Decision Process macroeconomic factors

2020-03-13abs ↗pdf ↗

Optimal Sample Complexity of Reinforcement Learning for Mixing Discounted Markov Decision Processescs.LG

New RL theory reduces sample complexity for mixing MDPs.

problem Optimal sample complexity for reinforcement learning in mixing MDPs.

method Regeneration-type ideas to analyze mixing times.

result Optimal sample complexity depends on mixing time, not just discount factor.

reinforcement learning Markov decision process mixing sample complexity regeneration

2023-02-15abs ↗pdf ↗

Robust

Q

-learning Algorithm for Markov Decision Processes under Wasserstein Uncertaintycs.LG

New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.

problem Solving Markov decision processes with uncertain transition probabilities.

method Distributionally robust

Q

-learning algorithm for Wasserstein uncertainty.

result Convergence of the algorithm proved and demonstrated with real data.

Q-learning Markov decision processes Wasserstein uncertainty distributional robustness

2022-09-30abs ↗pdf ↗

Model-Free Reinforcement Learning: from Clipped Pseudo-Regret to Sample Complexitycs.LG

Improved model-free RL algorithm with reduced sample complexity.

problem Learning optimal policies in discounted MDPs efficiently.

method Proposes new algorithms with improved sample complexity bounds.

result Achieves better sample complexity than existing methods.

Reinforcement Learning Model-Free Sample Complexity Markov Decision Process Pseudo-Regret

2020-06-06abs ↗pdf ↗

Improved Exploration in Factored Average-Reward MDPscs.LG

Improved exploration in factored average-reward MDPs reduces regret.

problem Minimizing regret in unknown Factored Markov Decision Processes (FMDPs).

method DBN-UCRL strategy, inspired by UCRL2, uses Bernstein-type confidence sets for individual elements of the transition function.

result Achieves a regret bound with a leading term strictly improving over existing bounds.

Factored Markov Decision Processes Average-reward criterion UCRL2 Bernstein-type confidence sets Regret minimization

2020-09-09abs ↗pdf ↗

Maximum Expected Hitting Cost of a Markov Decision Process and Informativeness of Rewardscs.LG

We propose a new complexity measure for Markov decision processes (MDPs), the maximum expected hitting cost (MEHC). This measure tightens the closely related notion of diameter [JOA10] by accounting for the reward structure. We show that this parameter replaces diameter in the upper bound on the optimal value span of a…

2019-07-03abs ↗pdf ↗

Nearly Minimax Optimal Reinforcement Learning for Discounted MDPscs.LG

UCBVI-γ algorithm minimizes regret in discounted MDPs.

problem Minimizing regret in discounted MDPs.

method Optimism in the face of uncertainty principle and Bernstein-type bonus.

result UCBVI-γ achieves nearly minimax optimal regret.

reinforcement learning Markov Decision Processes discounted MDPs UCBVI-γ regret minimization

2020-10-01abs ↗pdf ↗

A Unifying Framework for Reinforcement Learning and Planningcs.LG

This paper presents a unifying framework for reinforcement learning and planning.

problem Sequential decision making in AI, formalized as MDP optimization.

method A unifying algorithmic framework (FRAP) for reinforcement learning and planning.

result Identifies common dimensions in MDP planning and learning algorithms.

Reinforcement Learning Planning Markov Decision Process Artificial Intelligence Algorithmic Design

2020-06-26abs ↗pdf ↗

Asymptotically optimal regret in communicating Markov decision processescs.LG

Paper presents an algorithm for optimal regret in communicating Markov decision processes.

problem Achieving optimal regret in Markov decision processes with a communicating assumption.

method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.

result The algorithm achieves asymptotically optimal regret

K(M) \log(T) + \mathrm{o}(\log(T))

for communicating Markov decision processes.

Markov decision processes asymptotically optimal regret communicating assumption exploration

2025-05-23abs ↗pdf ↗

Thompson Sampling for Learning Parameterized Markov Decision Processesstat.ML

We consider reinforcement learning in parameterized Markov Decision Processes (MDPs), where the parameterization may induce correlation across transition probabilities or rewards. Consequently, observing a particular state transition might yield useful information about other, unobserved, parts of the MDP. We present a…

2014-06-29abs ↗pdf ↗

Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescs.LG

New RL algorithm for linear MDPs with nearly optimal regret.

problem Optimizing reinforcement learning for linear mixture Markov decision processes.

method Proposed a new Bernstein-type concentration inequality for self-normalized martingales and a computationally efficient algorithm UCRL-VTR+.

result UCRL-VTR+ achieves nearly minimax optimal regret of

ilde O(dH\sqrt{T})

reinforcement learning Markov decision processes linear function approximation minimax optimality Bernstein inequality

2020-12-15abs ↗pdf ↗

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision processeess.SY

Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.

problem Optimality in safety-constrained Markov decision processes with multichain structure.

method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.

result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.

Markov decision process Reinforcement learning Safety constraints Optimality Multi-objective optimization

2023-02-25abs ↗pdf ↗

Temporal Difference Learning as Gradient Splittingcs.LG

Temporal difference learning explained through gradient splitting, improving convergence times.

problem Learning value functions in Markov Decision Processes with linear approximations.

method Interpreting TD learning as gradient splitting and applying convergence proofs from gradient descent.

result Improved convergence times for TD learning, especially with a minor variation.

temporal difference learning gradient descent convergence Markov decision processes linear function approximation

2020-10-27abs ↗pdf ↗

Sufficient Markov Decision Processes with Alternating Deep Neural Networksstat.ME

Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…

2017-04-25abs ↗pdf ↗

Average optimality for risk-sensitive control with general state spaceq-fin.RM

This paper deals with discrete-time Markov control processes on a general state space. A long-run risk-sensitive average cost criterion is used as a performance measure. The one-step cost function is nonnegative and possibly unbounded. Using the vanishing discount factor approach, the optimality inequality and an optim…

2007-04-03abs ↗pdf ↗

Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingstat.ML

The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an…

2020-02-05abs ↗pdf ↗

An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processescs.LG

Develops an actor-critic algorithm for risk-sensitive Markov decision processes.

problem Risk-sensitive cost criterion in Markov decision processes.

method Actor-critic algorithm with function approximation.

result Asymptotic convergence of the actor-critic algorithm.

actor-critic Markov decision processes risk-sensitive function approximation asymptotic convergence

2025-02-17abs ↗pdf ↗

Markov Decision Processes with Risk-Sensitive Criteria: An Overviewq-fin.RM

Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.

problem Optimizing decision-making under risk in Markov processes.

method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.

result Conditions for the existence of optimal policies and solution procedures are provided.

Markov decision processes risk-sensitive Optimized Certainty Equivalent risk measures infinite horizon

2023-11-12abs ↗pdf ↗

Reinforcement Learning with Expert Trajectory For Quantitative Tradingcs.LG

Paper proposes a reinforcement learning method for trading using expert trajectories.

problem Inability of existing methods to handle long-term goals and delayed rewards in futures trading.

method Modeling futures trading as MDP, using reinforcement learning with expert trajectories and multiple short-term alpha factors.

result The proposed method outperforms traditional and deep learning methods in trading performance.

reinforcement learning quantitative trading expert trajectories Markov decision process futures trading

2021-05-09abs ↗pdf ↗

Regret and Sample Complexity of Online Q-Learning via Concentration of Stochastic Approximation with Time-Inhomogeneous Markov Chainscs.LG

Improved online Q-learning for MDPs with concentration bounds.

problem Online Q-learning in infinite-horizon discounted MDPs with sublinear regret for large gaps.

method Smoothed

ε_n

-Greedy exploration scheme combining

ε_n

-greedy and Boltzmann exploration, analyzed using concentration bounds for contractive Markovian stochastic approximation.

result Near-

ilde{O}(N^{9/10})

regret bound for Smoothed

ε_n

-Greedy exploration scheme.

online Q-learning Markov decision processes regret bounds stochastic approximation concentration inequalities

2026-02-18abs ↗pdf ↗

Off-Policy Risk Assessment in Markov Decision Processescs.LG

Paper develops a new estimator for MDPs' risk functionals with lower variance and bias.

problem Estimating the distribution of returns in MDPs with high variance and bias.

method Developed a doubly robust (DR) estimator for the CDF of returns in MDPs, incorporating model-based estimation to mitigate variance issues.

result The DR estimator achieves lower variance and bias compared to IS estimators, and matches minimax lower bounds.

Markov decision processes risk assessment doubly robust model-based estimation variance reduction

2022-09-21abs ↗pdf ↗

Max-Plus Matching Pursuit for Deterministic Markov Decision Processescs.LG

We consider deterministic Markov decision processes (MDPs) and apply max-plus algebra tools to approximate the value iteration algorithm by a smaller-dimensional iteration based on a representation on dictionaries of value functions. The setup naturally leads to novel theoretical results which are simply formulated due…

2019-06-20abs ↗pdf ↗

Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewardscs.LG

The paper develops a reinforcement learning model to estimate ad impact considering delayed and cumulative effects.

problem Accurately estimating ad impact considering delayed and long-term effects, cumulative impacts, and customer heterogeneity.

method Modeling ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards, proposing a two-stage maximum likelihood estimator and reinforcement learning algorithm.

result Achieves a near-optimal regret bound of

\tilde{O}{(dH^2\sqrt{T})}

, validating the approach through simulation experiments.

reinforcement learning ad impact estimation delayed rewards customer heterogeneity contextual Markov decision process

2025-10-22abs ↗pdf ↗

Risk-Averse Markov Decision Processes through a Distributional Lensmath.OC

Risk measures applied to dynamic Markov processes with varying risk aversion.

problem Investigating dynamic risk measures in Markov decision processes with varying risk aversion.

method Distributional viewpoint on law-invariant convex risk measures, applied to Markov decision processes with latent costs and random actions.

result Existence of optimal policies in finite and infinite time horizons under mild assumptions.

risk measures Markov decision processes dynamic programming risk aversion convex risk measures

2022-03-17abs ↗pdf ↗

State Action Separable Reinforcement Learningcs.LG

A new RL paradigm reduces state-action-value function approximation inefficiency.

problem Challenges in state-action-value function approximation for RL.

method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.

result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.

Reinforcement Learning State Action Separable Value Function Markov Decision Process Efficiency

2020-06-05abs ↗pdf ↗

Inverse Risk-Sensitive Reinforcement Learningcs.LG

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology, behavioral economics, an…

2017-03-29abs ↗pdf ↗

Bayesian Inverse Reinforcement Learning for Collective Animal Movementcs.LG

Bayesian method infers local rules for collective animal movement.

problem Learn local rules governing long-term group behaviors.

method Bayesian Inverse Reinforcement Learning with Linearly-Solvable Markov Decision Process.

result Recover true costs and find value of collective movement.

Bayesian Inverse Reinforcement Learning Collective Animal Movement Linearly-Solvable Markov Decision Process Agent-based Models Markov Decision Process

2020-09-08abs ↗pdf ↗

Batch Policy Learning in Average Reward Markov Decision Processesmath.ST

The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.

problem Maximizing long-term average reward in Markov Decision Processes with batch learning.

method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.

result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.

Markov Decision Processes Average Reward Batch Learning Doubly Robust Estimator Optimization Algorithm

2020-07-23abs ↗pdf ↗

Lattice investment projects support process model with corruptionq-fin.MF

Lattice investment projects support process model with corruption is formulated and analyzed. The model is based on the Ising lattice model of ferromagnetic but takes deal with the social phenomenon. Set of corruption agents is considered. It is supposed that agents are placed in sites of the lattice. Agents take decis…

2019-01-25abs ↗pdf ↗

Navigating to the Best Policy in Markov Decision Processesstat.ML

New algorithm identifies best policy in MDPs faster.

problem Identifying the best policy in Markov Decision Processes.

method Problem-dependent lower bound and first algorithm with instance-specific sample complexity.

result First algorithm with reduced exploration rate for faster convergence.

Markov Decision Processes Pure Exploration Ergodicity Non-homogeneous Markov Chains

2021-06-05abs ↗pdf ↗

Model-based Multi-Agent Reinforcement Learning with Cooperative Prioritized Sweepingcs.LG

We present a new model-based reinforcement learning algorithm, Cooperative Prioritized Sweeping, for efficient learning in multi-agent Markov decision processes. The algorithm allows for sample-efficient learning on large problems by exploiting a factorization to approximate the value function. Our approach only requir…

2020-01-15abs ↗pdf ↗

Markov cubature rules for polynomial processesmath.PR

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using …

2017-07-21abs ↗pdf ↗

Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPscs.LG

Study examines sample complexity for RL with safety constraints.

problem Ensuring safety in RL for systems with constraints.

method Analyzes two RL approaches: generative and online.

result Sample complexity increased by a logarithmic factor in constraints.

Reinforcement Learning Safety Constraints Sample Complexity Constrained Markov Decision Process PAC Learning

2020-08-01abs ↗pdf ↗

Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecs.LG

CPPO learns policies from partial offline data in MDPs with structural assumptions.

problem Offline Reinforcement Learning with partial coverage assumption.

method Constrained Pessimistic Policy Optimization (CPPO) using a function class and model class constraint.

result CPPO achieves PAC guarantee with partial coverage, learning competitive policies.

Offline Reinforcement Learning Model-based Learning Partial Coverage PAC Guarantee Constrained Policy Optimization

2021-07-13abs ↗pdf ↗

Distributionally Robust Markov Decision Processes and their Connection to Risk Measuresmath.OC

We develop robust Markov Decision Processes with risk measures for uncertain environments.

problem Uncertainty in Markov Decision Processes and its impact on risk measures.

method Formulation as a Stackelberg game, robust cost and value iterations, existence of optimal policies.

result Existence of deterministic optimal policies for robust optimization and risk measures.

Markov Decision Processes Robust Optimization Risk Measures Uncertainty Stackelberg Game

2020-07-26abs ↗pdf ↗

Reinforcement Learning in Factored MDPs: Oracle-Efficient Algorithms and Tighter Regret Bounds for the Non-Episodic Settingstat.ML

We study reinforcement learning in non-episodic factored Markov decision processes (FMDPs). We propose two near-optimal and oracle-efficient algorithms for FMDPs. Assuming oracle access to an FMDP planner, they enjoy a Bayesian and a frequentist regret bound respectively, both of which reduce to the near-optimal bound …

2020-02-06abs ↗pdf ↗

Concentration bounds for SSP Q-learning for average cost MDPscs.LG

Q-learning for average cost MDPs gets a concentration bound.

problem Finding bounds for Q-learning in average cost MDPs.

method Derives a concentration bound using shortest path problem equivalence.

result Numerical comparison with relative value iteration shows the bound's effectiveness.

Q-learning Markov decision processes concentration bounds shortest path problem

2022-06-07abs ↗pdf ↗