On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “robust MDPs”

Robust Markov Decision Processes without Model Estimationstat.ML

New method removes oracle and reduces memory usage for robust MDPs.

problem Applying robust MDPs in practice due to model estimation and oracle requirements.

method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.

result Sample-efficient algorithm with lower storage requirement and no oracle.

Markov Decision Processes Robustness Stochastic Gradient Model-Free Oracle

2023-02-02abs ↗pdf ↗

Scaling Up Robust MDPs by Reinforcement Learningcs.LG

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…

2013-06-26abs ↗pdf ↗

Partial Policy Iteration for L1-Robust Markov Decision Processescs.LG

New algorithms solve robust MDPs efficiently, significantly faster than existing methods.

problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.

method Partial policy iteration and fast robust Bellman operator computation methods.

result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.

robust MDPs partial policy iteration fast computation L1-norms quasi-linear time

2020-06-16abs ↗pdf ↗

Online Policy Optimization for Robust MDPcs.LG

New RL algorithm tackles online robust MDPs with uncertainty.

problem Developing robust reinforcement learning models for real-world environments.

method Proposes a robust optimistic policy optimization algorithm for online robust MDPs.

result Establishes the first regret bound for online robust MDPs.

reinforcement learning robust MDP online policy optimization optimistic policy regret bound

2022-09-28abs ↗pdf ↗

Distributional Robustness and Regularization in Reinforcement Learningmath.OC

New approach connects reinforcement learning robustness and regularization.

problem Dealing with external uncertainty in reinforcement learning.

method Introducing Wasserstein distributionally robust MDPs and a new regularizer.

result Established a dual relation between robust MDPs and regularization.

reinforcement learning distributional robustness regularization MDP external uncertainty

2020-03-05abs ↗pdf ↗

Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticsstat.ML

Paper studies robust MDPs, improving sample complexity and asymptotic performance.

problem Optimal robust policy and value function in robust MDPs with generative models.

method Improves prior results on non-asymptotic and asymptotic performances of robust MDPs, considering various uncertainty sets.

result Improved sample complexity and asymptotic normality of optimal robust value function.

robust MDPs sample complexity asymptotic performance uncertainty sets optimal policy

2021-05-09abs ↗pdf ↗

Learning Robust State Abstractions for Hidden-Parameter Block MDPscs.LG

This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.

problem Limited observability of state in HiP-MDPs for real-world scenarios with rich observation spaces.

method Inspired by Block MDPs, the work extends HiP-MDPs to enable robust state abstractions for multi-task and meta-reinforcement learning.

result Transfer and generalization bounds based on task and state similarity, and sample complexity bounds that depend on the aggregate number of samples across tasks.

Robust State Abstractions Hidden-Parameter Markov Decision Processes Block Markov Decision Processes Multi-task Reinforcement Learning Meta-reinforcement Learning

2020-07-14abs ↗pdf ↗

Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approachmath.OC

Develops robust MDPs for unknown disturbances with performance guarantees.

problem Unknown disturbance distribution in MDPs.

method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.

result Robust optimal value function converges to true optimal value function with increasing sample sizes.

Markov decision processes robust optimization unknown disturbances concentration inequalities performance guarantees

2026-03-09abs ↗pdf ↗

Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscs.LG

Deep RL policies share adversarial features across different MDPs.

problem Understanding decision boundaries and loss landscapes in neural policies.

method Investigating similarities in high sensitivity directions across MDPs using Arcade Learning Environment.

result High sensitivity directions for neural policies are correlated across MDPs, suggesting shared non-robust features.

deep reinforcement learning neural policies MDPs adversarial features decision boundaries

2021-12-16abs ↗pdf ↗

Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learningcs.LG

Study improves reinforcement learning for stable long-term performance.

problem Distributionally robust average-reward reinforcement learning for stable long-term performance.

method Proposes two algorithms to achieve near-optimal sample complexity.

result Achieves a sample complexity of

O(|\mathbf{S}||\mathbf{A}| t_{\mathrm{mix}}^2\varepsilon^{-2})

for estimating optimal policy and robust average reward.

reinforcement learning distributionally robust average-reward sample complexity stability

2025-05-15abs ↗pdf ↗

Non-Rectangular Average-Reward Robust MDPs: Optimal Policies and Their Transient Valuesmath.OC

Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.

problem Non-rectangular robust Markov decision processes under average-reward criterion.

method Proves history-dependent policies are robust-optimal, introduces transient-value framework, constructs epoch-based policy.

result Existence and properties of robust optimal policies, transient value bounds.

robust MDPs average-reward transient values optimal policies stationary kernel

2026-03-01abs ↗pdf ↗

Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapstat.ML

Improved off-policy evaluation for MDPs with weak distributional overlap.

problem Evaluation of policies when target and data-collection distributions are not strongly overlapping.

method Truncated Doubly Robust (TDR) estimators for off-policy evaluation in MDPs under weak distributional overlap.

result TDR estimators can recover large-sample behavior and are consistent even when distribution ratios are not square-integrable.

off-policy evaluation Markov decision processes doubly robust distributional overlap weak overlap

2024-02-13abs ↗pdf ↗

Reinforcement Learning under Model Mismatchcs.LG

We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs to the model-free Reinforcement Learning setting, where we do not have access to the mod…

2017-06-15abs ↗pdf ↗

Bayesian regularization of empirical MDPscs.LG

Bayesian regularization improves policy performance in noisy MDPs.

problem Suboptimal policies from estimated model parameters.

method Bayesian regularization of MDP objective function with prior information.

result Regularized policies show better robustness against model noise.

Bayesian regularization Markov decision processes policy optimization robustness noise

2022-08-03abs ↗pdf ↗

Tight Bayesian Ambiguity Sets for Robust MDPscs.LG

Robustness is important for sequential decision making in a stochastic dynamic environment with uncertain probabilistic parameters. We address the problem of using robust MDPs (RMDPs) to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution is det…

2018-11-15abs ↗pdf ↗

Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscs.LG

Robust MDPs (RMDPs) can be used to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution are determined by the ambiguity set---the set of plausible transition probabilities---which is usually constructed as a multi-dimensional confidence region. E…

2019-02-20abs ↗pdf ↗

A Model Selection Approach for Corruption Robust Reinforcement Learningcs.LG

New model selects robustly in adversarial reinforcement learning with unknown corruption.

problem Adversarial corruption in reinforcement learning with unknown total corruption amount.

method Model selection approach for finite-horizon tabular and linear MDPs.

result First worst-case optimal bound without knowledge of total corruption.

reinforcement learning corruption robustness model selection adversarial tabular MDPs

2021-10-07abs ↗pdf ↗

Provably Efficient Sample Complexity for Robust CMDPcs.LG

Paper tackles robust MDPs with sample complexity guarantees.

problem Learning safe policies in uncertain environments.

method Introduces augmented state space and RCVI algorithm.

result First sample complexity guarantee for RCMDPs.

Robust MDP Sample Complexity Policy Optimization Value Iteration Safety Constraints

2025-11-10abs ↗pdf ↗

Optimizing Percentile Criterion Using Robust MDPscs.LG

We address the problem of computing reliable policies in reinforcement learning problems with limited data. In particular, we compute policies that achieve good returns with high confidence when deployed. This objective, known as the \emph{percentile criterion}, can be optimized using Robust MDPs~(RMDPs). RMDPs general…

2019-10-23abs ↗pdf ↗

Minimax Optimal Variance-Aware Regret Bounds for Multinomial Logistic MDPscs.AI

Improved regret bound for MNL MDPs with variance-aware approach.

problem Optimal reinforcement learning for MNL MDPs with structured variance.

method Introducing a problem-dependent constant measuring average variance, proposing an algorithm with improved regret bound.

result Minimax optimal regret bound of

O(dH^2\barσ_T\sqrt{T})

for structured MDPs.

Reinforcement Learning Multinomial Logistic MDPs Regret Bounds Variance-Aware Logistic Bandits

2026-05-19abs ↗pdf ↗

Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationaritystat.ML

New method for estimating and optimizing MDPs without stationarity.

problem Challenges in offline contextual MDP estimation without stationarity.

method Introduces a new adaptive estimation and cost optimization approach for contextual MDPs.

result First robust, theoretically backed method for offline contextual MDP estimation.

Contextual MDPs Offline Estimation Optimal Control Non-stationarity T-Estimation

2026-05-05abs ↗pdf ↗

Robust Batch Policy Learning in Markov Decision Processesmath.ST

The paper tackles robust policy learning in MDPs using statistical methods.

problem Offline data-driven sequential decision making in MDPs.

method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.

result Established a rate-optimal regret bound up to a logarithmic factor.

Markov Decision Processes Offline Policy Learning Statistical Efficiency Robust Policy Semi-parametric Statistics

2020-11-09abs ↗pdf ↗

Optimizing Norm-Bounded Weighted Ambiguity Sets for Robust MDPscs.LG

Optimal policies in Markov decision processes (MDPs) are very sensitive to model misspecification. This raises serious concerns about deploying them in high-stake domains. Robust MDPs (RMDP) provide a promising framework to mitigate vulnerabilities by computing policies with worst-case guarantees in reinforcement learn…

2019-12-04abs ↗pdf ↗

Robust Transfer Learning with Side Informationstat.ML

Improves robust transfer learning with side information.

problem Addressing environmental shift in MDPs with side information.

method Estimate-centered uncertainty sets with side information integration.

result Improved robust policy with reduced sub-optimality gap.

transfer learning robust optimization side information MDPs environment shift

2026-03-09abs ↗pdf ↗

Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processesstat.ML

We introduce a new formulation of the Hidden Parameter Markov Decision Process (HiP-MDP), a framework for modeling families of related tasks using low-dimensional latent embeddings. Our new framework correctly models the joint uncertainty in the latent parameters and the state space. We also replace the original Gaussi…

2017-06-20abs ↗pdf ↗

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescs.AI

We develop efficient and sharp bounds on policy value under perturbations in MDPs.

problem Evaluating policies under best- and worst-case perturbations in MDPs with transition observations.

method Proposed a perturbation model for MDPs, developed semiparametrically efficient estimator with asymptotic normality.

result Semiparametrically efficient and asymptotically normal estimator for policy value bounds.

Markov decision processes perturbation model policy evaluation robustness inference

2024-03-29abs ↗pdf ↗

Non-Stationary Dividend-Price Ratiosq-fin.PM

Dividend yields have been widely used in previous research to relate stock market valuations to cash flow fundamentals. However, this approach relies on the assumption that dividend yields are stationary. Due to the failure to reject the hypothesis of a unit root in the classical dividend-price ratio for the US stock m…

2019-02-16abs ↗pdf ↗

Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processesstat.ME

Paper tackles robust decision-making from multiple sites with shared structure.

problem Learning robust sequential decisions from heterogeneous multi-site datasets.

method Group-Robust MDPs with d-rectangular uncertainty sets, feature-wise worst-case aggregation, and cluster-level pooling.

result Proves suboptimality bound for robust planning policy under robust partial coverage assumption.

Sequential Decisions Multi-Site Learning Robust MDPs Group-Robust Heterogeneous Data

2026-02-02abs ↗pdf ↗

Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisionscs.LG

Detects adversarial directions to make reinforcement learning policies more robust.

problem Adversarial attacks exploit non-robust directions in reinforcement learning policies, leading to instability.

method Local quadratic approximation of deep neural policy loss to identify non-robust directions.

result Provides a theoretical basis for distinguishing safe from adversarial observations.

reinforcement learning adversarial attacks policy instability deep neural networks local quadratic approximation

2023-06-09abs ↗pdf ↗

An Instrumental Variable Approach to Confounded Off-Policy Evaluationstat.ML

Paper develops IV method for consistent OPE in confounded MDPs.

problem Confounding variables in observational data affect OPE effectiveness.

method Instrumental variable approach for consistent off-policy evaluation.

result IV method enables correct identification of target policy's value.

Instrumental Variables Off-Policy Evaluation Confounded MDPs Markov Decision Processes Consistent Estimation

2022-12-29abs ↗pdf ↗

Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincs.LG

New bounds assess policy evaluation under unobserved confounders, showing model-based methods are more effective.

problem Policy evaluation under unobserved confounders in uncertain causal environments.

method Developed worst-case bounds for sensitivity to unobserved confounders, demonstrating model-based methods are more effective.

result Model-based approaches with robust MDPs provide sharper lower bounds for policy evaluation.

policy evaluation uncertain causality confounders model-based methods robust MDPs

2022-04-02abs ↗pdf ↗

Markov Decision Processes with Recursive Risk Measuresmath.OC

The paper studies risk-sensitive MDPs with recursive risk measures.

problem Risk-sensitive decision-making in MDPs with unbounded costs.

method Recursive application of static risk measures, Bellman equation derivation, existence of optimal policies.

result Existence of Markovian optimal policies for infinite planning horizons, contractive model for stationary optimal policy.

Markov Decision Processes Recursive Risk Measures Bellman Equation Optimal Policies Stationary Policies

2020-10-14abs ↗pdf ↗

Is RLHF More Difficult than Standard RL?cs.LG

Paper shows RLHF can be solved similarly to standard RL.

problem Difficulty of RLHF compared to standard RL.

method Reduction to reward-based RL techniques.

result RLHF can be solved using existing algorithms for reward-based RL.

Reinforcement Learning Human Feedback Reward-based RL Preference-based RL Robust RL

2023-06-25abs ↗pdf ↗

Capturing Financial markets to apply Deep Reinforcement Learningq-fin.CP

In this paper we explore the usage of deep reinforcement learning algorithms to automatically generate consistently profitable, robust, uncorrelated trading signals in any general financial market. In order to do this, we present a novel Markov decision process (MDP) model to capture the financial trading markets. We r…

2019-07-09abs ↗pdf ↗

Off-Policy Risk Assessment in Markov Decision Processescs.LG

Paper develops a new estimator for MDPs' risk functionals with lower variance and bias.

problem Estimating the distribution of returns in MDPs with high variance and bias.

method Developed a doubly robust (DR) estimator for the CDF of returns in MDPs, incorporating model-based estimation to mitigate variance issues.

result The DR estimator achieves lower variance and bias compared to IS estimators, and matches minimax lower bounds.

Markov decision processes risk assessment doubly robust model-based estimation variance reduction

2022-09-21abs ↗pdf ↗

Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learningcs.LG

NVMDP framework tackles non-stationary MDPs with varying discount rates.

problem Challenges in non-stationary environments and infinite-horizon formulations for reinforcement learning.

method Introduces NVMDP framework that accommodates non-stationarity and varying discount rates.

result NVMDPs provide a flexible mechanism to shape optimal policies without altering state or action spaces.

Reinforcement Learning Markov Decision Processes Non-stationary Varying Discounting Optimal Policy

2025-11-17abs ↗pdf ↗

Optimizing for the Future in Non-Stationary MDPscs.LG

Prognosticator improves performance in non-stationary MDPs.

problem Real-world applications often violate the stationary assumption in reinforcement learning.

method A policy gradient algorithm that maximizes a forecast of future performance.

result Prognosticator outperforms online adaptation techniques in non-stationary MDPs.

reinforcement learning non-stationary MDPs policy gradient forecast performance maximization

2020-05-17abs ↗pdf ↗

Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPscs.LG

Adaptive reduction scheme approximates optimal policy in regularized MDPs.

problem Finding near optimal policy in regularized MDPs with biased solutions.

method Adaptive reduction of regularization parameter λ to approximate optimal policy.

result Iteration complexity reduced for obtaining ε-optimal policy.

MDPs Regularization Policy approximation Adaptive reduction Optimization

2020-10-31abs ↗pdf ↗

Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learningstat.ML

We study Exo-MDPs to reduce sample complexity in reinforcement learning.

problem Reducing sample complexity in reinforcement learning for structured MDPs.

method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.

result Proved

O(H^{3/2}d\sqrt{K})

regret bound for Exo-MDPs, matching lower bounds.

Reinforcement Learning Markov Decision Processes Exogenous States Sample-Efficient Inventory Control

2024-09-22abs ↗pdf ↗

Simultaneously Learning Stochastic and Adversarial Episodic MDPs with Known Transitioncs.LG

Algorithm learns both stochastic and adversarial MDPs with best-of-both-worlds guarantees.

problem Learning episodic MDPs with known transition and bandit feedback.

method Follow-the-Regularized-Leader method with a hybrid regularizer.

result Achieves

\mathcal{O}(log T)

regret for stochastic losses and

ilde{\mathcal{O}}(\sqrt{T})

regret for adversarial losses.

MDP Bandit Episodic Regret Adversarial

2020-06-10abs ↗pdf ↗

Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcs.LG

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…

2017-09-19abs ↗pdf ↗

Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcs.LG

Paper establishes new lower bounds for MDPs with changing transition kernels.

problem Minimizing sample complexity and regret in non-stationary MDPs.

method Developed novel lower bounds and constructed hard MDPs.

result Proved

Ω((H^3SA/ε^2)\log(1/δ))

sample complexity lower bound.

Reinforcement Learning MDP Non-stationary Lower Bounds PAC

2020-10-07abs ↗pdf ↗

Solving Continual Combinatorial Selection via Deep Reinforcement Learningcs.LG

We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present …

2019-09-09abs ↗pdf ↗

Efficient

Q

-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learningcs.LG

New algorithms improve robust reinforcement learning under uncertainty.

problem Robust reinforcement learning in MDPs with contamination.

method Non-asymptotic convergence analysis of

Q

-learning and actor-critic methods.

result Efficient algorithms learn robust policies with minimal samples.

reinforcement learning robustness Q-learning actor-critic Markov decision processes

2025-06-08abs ↗pdf ↗

Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncs.LG

This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded ev…

2019-04-22abs ↗pdf ↗

Regret and Sample Complexity of Online Q-Learning via Concentration of Stochastic Approximation with Time-Inhomogeneous Markov Chainscs.LG

Improved online Q-learning for MDPs with concentration bounds.

problem Online Q-learning in infinite-horizon discounted MDPs with sublinear regret for large gaps.

method Smoothed

ε_n

-Greedy exploration scheme combining

ε_n

-greedy and Boltzmann exploration, analyzed using concentration bounds for contractive Markovian stochastic approximation.

result Near-

ilde{O}(N^{9/10})

regret bound for Smoothed

ε_n

-Greedy exploration scheme.

online Q-learning Markov decision processes regret bounds stochastic approximation concentration inequalities

2026-02-18abs ↗pdf ↗

Optimality Guarantees for Particle Belief Approximation of POMDPscs.AI

New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.

problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.

method Bounding particle filtering error and adapting MDP algorithms to POMDPs.

result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.

POMDPs Particle Belief Approximation MDPs Error Bounds Continuous Spaces

2022-10-10abs ↗pdf ↗

Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcs.LG

New algorithm reduces performance loss in IRL with mismatched transition dynamics.

problem Performance degradation in inverse reinforcement learning due to mismatched transition dynamics.

method Proposed a robust Maximum Causal Entropy (MCE) IRL algorithm leveraging robust reinforcement learning insights.

result Empirically demonstrated stable performance improvement under transition dynamics mismatches.

Inverse Reinforcement Learning Transition Dynamics Mismatch Maximum Causal Entropy Robust Reinforcement Learning

2020-07-02abs ↗pdf ↗