On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “MDP framework”

On learning history based policies for controlling Markov decision processescs.LG

Paper analyzes history-based RL methods for MDPs, introduces a theoretical framework and practical algorithm.

problem Improving RL performance in MDPs using history-based features.

method Theoretical framework for history-based RL, practical algorithm design.

result Practical RL algorithm shows effectiveness on continuous control tasks.

Reinforcement Learning Markov Decision Processes History-based Features Partially Observable MDP Theoretical Analysis

2022-11-06abs ↗pdf ↗

Efficiently Solving MDPs with Stochastic Mirror Descentcs.LG

Unified framework for solving MDPs with stochastic mirror descent.

problem Approximately solving infinite-horizon Markov decision processes (MDPs).

method Primal-dual stochastic mirror descent for MDPs with a unified framework.

result Computes ε-optimal policies with expected samples for both average-reward and discounted MDPs.

Markov decision processes stochastic mirror descent MDP solving approximation algorithms reinforcement learning

2020-08-28abs ↗pdf ↗

A General Markov Decision Process Framework for Directly Learning Optimal Control Policiescs.LG

We consider a new form of reinforcement learning (RL) that is based on opportunities to directly learn the optimal control policy and a general Markov decision process (MDP) framework devised to support these opportunities. Derivations of general classes of our control-based RL methods are presented, together with form…

2019-05-28abs ↗pdf ↗

Problem Dependent Reinforcement Learning Bounds Which Can Identify Bandit Structure in MDPscs.LG

New algorithm matches best regret bound for tabular Contextual Bandit problems.

problem Learning from observations in uncertain environments.

method A minor variant of a reinforcement learning algorithm for MDPs.

result The algorithm matches the best possible regret bound

ilde O (\sqrt{SAT})

for tabular Contextual Bandit problems.

Reinforcement Learning Bandit Problems MDPs Regret Bound Contextual Bandits

2019-11-03abs ↗pdf ↗

A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcs.LG

A new framework reduces RL sample complexity for complex MDPs.

problem Handling large state and action spaces in reinforcement learning.

method Unified model-based and model-free RL framework with ABC class, novel estimation function, and functional eluder dimension.

result OPERA algorithm achieves sample-efficient regret bounds for various MDP models.

reinforcement learning function approximation sample efficiency Markov Decision Processes model-based

2022-09-30abs ↗pdf ↗

DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscs.LG

DeepAveragers solves offline RL by solving derived MDPs from static data.

problem Offline reinforcement learning with limited data.

method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.

result The approach can lower-bound performance and scale to complex offline RL problems.

offline reinforcement learning non-parametric MDPs deep representations costs performance bounds

2020-10-18abs ↗pdf ↗

MOReL : Model-Based Offline Reinforcement Learningcs.LG

MOReL learns offline RL policies using pessimistic MDPs.

problem Offline RL's data efficiency and velocity.

method Two-step process: learn P-MDP and near-optimal policy in it.

result MOReL is minimax optimal and matches state-of-the-art results.

Offline RL Model-based RL Pessimistic MDP Minimax optimality Policy evaluation

2020-05-12abs ↗pdf ↗

Reward is enough for convex MDPscs.AI

Reward suffices for convex MDPs, expanding RL to new problems.

problem Capturing goals as convex functions of stationary distribution.

method Reformulated as a min-max game using Fenchel duality.

result Convex MDPs require non-stationary reward functions.

reinforcement learning Markov decision processes convex optimization Fenchel duality min-max games

2021-06-01abs ↗pdf ↗

Demystifying Linear MDPs and Novel Dynamics Aggregation Frameworkstat.ML

We prove a lower bound for feature dimension in linear MDPs and propose a novel dynamics aggregation framework.

problem The limitation of feature dimension in linear MDPs and the need for efficient hierarchical reinforcement learning.

method We propose a novel dynamics aggregation framework based on structural dynamics and design a provably efficient hierarchical reinforcement learning algorithm.

result Our algorithm achieves a regret of

ilde{O} ( d_ψ^{3/2} H^{3/2}\sqrt{ N T} )

and meets the condition

d_ψ^3 N \ll d^{3}

in most real-world environments.

linear MDPs feature dimension dynamics aggregation hierarchical reinforcement learning provably efficient

2024-10-31abs ↗pdf ↗

Making Linear MDPs Practical via Contrastive Representation Learningcs.LG

New method tackles MDPs by learning normalized representations efficiently.

problem Curse of dimensionality in MDPs.

method Contrastive representation learning for linear MDPs.

result First practical method with strong theoretical guarantees and empirical performance.

Markov decision processes linear MDPs representation learning contrastive estimation sample efficiency

2022-07-14abs ↗pdf ↗

Real-Time Reinforcement Learningcs.LG

New RL framework improves real-time control performance.

problem Real-time RL systems assume static states, leading to suboptimal outcomes.

method Introduces a new real-time RL framework where states and actions evolve simultaneously.

result RTAC algorithm outperforms existing state-of-the-art algorithms in real-time and non-real-time settings.

Reinforcement Learning Real-time Actor-Critic Soft Actor-Critic Markov Decision Processes

2019-11-11abs ↗pdf ↗

Generalized Linear Markov Decision Processstat.ML

Extends linear MDP to handle nonlinear rewards.

problem Restrictive linear MDP assumption limits real-world applicability.

method Proposes Generalized Linear MDP (GLMDP) with GLMs for rewards.

result Develops offline RL algorithms achieving suboptimality guarantees.

reinforcement learning Markov decision process generalized linear model nonlinear rewards sample efficiency

2025-06-01abs ↗pdf ↗

A Regularized Approach to Sparse Optimal Policy in Reinforcement Learningstat.ML

We propose and study a general framework for regularized Markov decision processes (MDPs) where the goal is to find an optimal policy that maximizes the expected discounted total reward plus a policy regularization term. The extant entropy-regularized MDPs can be cast into our framework. Moreover, under our framework, …

2019-03-02abs ↗pdf ↗

Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processesstat.ML

We introduce a new formulation of the Hidden Parameter Markov Decision Process (HiP-MDP), a framework for modeling families of related tasks using low-dimensional latent embeddings. Our new framework correctly models the joint uncertainty in the latent parameters and the state space. We also replace the original Gaussi…

2017-06-20abs ↗pdf ↗

Online Policy Optimization for Robust MDPcs.LG

New RL algorithm tackles online robust MDPs with uncertainty.

problem Developing robust reinforcement learning models for real-world environments.

method Proposes a robust optimistic policy optimization algorithm for online robust MDPs.

result Establishes the first regret bound for online robust MDPs.

reinforcement learning robust MDP online policy optimization optimistic policy regret bound

2022-09-28abs ↗pdf ↗

Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscs.LG

Deep RL policies share adversarial features across different MDPs.

problem Understanding decision boundaries and loss landscapes in neural policies.

method Investigating similarities in high sensitivity directions across MDPs using Arcade Learning Environment.

result High sensitivity directions for neural policies are correlated across MDPs, suggesting shared non-robust features.

deep reinforcement learning neural policies MDPs adversarial features decision boundaries

2021-12-16abs ↗pdf ↗

Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcs.LG

In this paper, we present a new class of Markov decision processes (MDPs), called Tsallis MDPs, with Tsallis entropy maximization, which generalizes existing maximum entropy reinforcement learning (RL). A Tsallis MDP provides a unified framework for the original RL problem and RL with various types of entropy, includin…

2019-01-31abs ↗pdf ↗

Fast Online Exact Solutions for Deterministic MDPs with Sparse Rewardscs.LG

Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision making under uncertainty. The classical approaches for solving MDPs are well known and have been widely studied, some of which rely on approximation techniques to solve MDPs with large state space and/or action space. However…

2018-05-08abs ↗pdf ↗

Reinforcement Learning from Adversarial Preferences in Tabular MDPscs.LG

New framework for reinforcement learning with adversarial preferences in tabular MDPs.

problem Learning from preferences rather than direct rewards in MDPs with adversarial settings.

method Developed PbMDPs framework, established lower bounds, and proposed algorithms for regret minimization.

result Achieved regret bounds of

Ω( (H^2 S K)^{1/3} T^{2/3} )

for PbMDPs with Borda scores.

Reinforcement Learning MDPs Adversarial Preferences Borda Scores Regret Minimization

2025-07-15abs ↗pdf ↗

Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learningcs.LG

NVMDP framework tackles non-stationary MDPs with varying discount rates.

problem Challenges in non-stationary environments and infinite-horizon formulations for reinforcement learning.

method Introduces NVMDP framework that accommodates non-stationarity and varying discount rates.

result NVMDPs provide a flexible mechanism to shape optimal policies without altering state or action spaces.

Reinforcement Learning Markov Decision Processes Non-stationary Varying Discounting Optimal Policy

2025-11-17abs ↗pdf ↗

MetaCURL: Non-stationary Concave Utility Reinforcement Learningcs.LG

MetaCURL tackles non-stationary MDPs with optimal dynamic regret.

problem Online learning in non-stationary Markov decision processes.

method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.

result Achieves optimal dynamic regret without prior knowledge of MDP changes.

Reinforcement Learning Non-stationary MDPs Meta-algorithm Optimal Regret Sleeping Expert

2024-05-30abs ↗pdf ↗

Scalable Online Exploration via Coverabilitycs.LG

New $L_1$ -Coverage objective simplifies exploration in reinforcement learning.

problem Challenges in exploration for high-dimensional domains.

method Introduces

L_1

-Coverage objective to enable efficient exploration and planning.

result First computationally efficient algorithms for online reinforcement learning with low coverability.

reinforcement learning exploration coverability policy optimization online learning

2024-03-11abs ↗pdf ↗

Comparison and Unification of Three Regularization Methods in Batch Reinforcement Learningcs.LG

This paper unifies three regularization methods in batch reinforcement learning.

problem Learning overly-complex models in batch reinforcement learning.

method Unified weighted average transition matrix framework for three regularization methods.

result Empirical evaluation confirms intuitions about regularization methods' performance.

reinforcement learning batch learning regularization Markov decision processes state-action pairs

2021-09-16abs ↗pdf ↗

Exploration for Multi-task Reinforcement Learning with Deep Generative Modelscs.AI

Exploration in multi-task reinforcement learning is critical in training agents to deduce the underlying MDP. Many of the existing exploration frameworks such as $E^3$ , $R_{max}$ , Thompson sampling assume a single stationary MDP and are not suitable for system identification in the multi-task setting. We present a nove…

2016-11-29abs ↗pdf ↗

Homomorphic Mappings for Value-Preserving State Aggregation in Markov Decision Processescs.LG

This paper optimizes MDP policies for efficient state aggregation.

problem Optimizing policies in aggregated Markov chains while preserving optimal performance.

method Homomorphic mappings to establish optimal policy equivalence and derive performance bounds.

result Developed HPG and EBHPG methods for efficient aggregation and policy optimization.

Markov Decision Processes State Aggregation Optimal Policy Equivalence Homomorphic Mappings Policy Gradient

2025-10-11abs ↗pdf ↗

Transfer Learning Across Patient Variations with Hidden Parameter Markov Decision Processesstat.ML

Due to physiological variation, patients diagnosed with the same condition may exhibit divergent, but related, responses to the same treatments. Hidden Parameter Markov Decision Processes (HiP-MDPs) tackle this transfer-learning problem by embedding these tasks into a low-dimensional space. However, the original formul…

2016-12-01abs ↗pdf ↗

Revisiting Weighted Strategy for Non-stationary Parametric Bandits and MDPscs.LG

This paper refines the weighted strategy for non-stationary parametric bandits and MDPs, improving regret bounds.

problem Non-stationary environments with gradual drifting patterns.

method Refined analysis framework for the weighted strategy, leading to simpler and more efficient algorithms.

result Improved regret bounds for linear bandits, generalized linear bandits, and self-concordant bandits.

non-stationary parametric bandits weighted strategy regret analysis linear bandits

2026-01-03abs ↗pdf ↗

Optimizing Norm-Bounded Weighted Ambiguity Sets for Robust MDPscs.LG

New method optimizes ambiguity sets for robust MDPs, improving policy robustness.

problem Sensitivity of optimal policies to model misspecification in MDPs.

method Proposes a new approach to optimize the shape of ambiguity sets, departing from norm-bounded uniform and symmetric sets.

result Empirical results show practical promise of weighted near-optimal ambiguity sets.

robust MDPs ambiguity sets Markov decision processes reinforcement learning model misspecification

2019-12-04abs ↗pdf ↗

Nearly Optimal Latent State Decoding in Block MDPscs.LG

We decode latent states in Block MDPs and learn near-optimal policies.

problem Model estimation and reward-free learning in Block MDPs.

method Information-theoretical lower bound and efficient model estimation algorithm.

result Our algorithm approaches the information-theoretical limit for latent state decoding and converges to optimal policies.

latent state decoding Block MDPs reward-free learning model estimation near-optimal policies

2022-08-17abs ↗pdf ↗

Domain Adaptive Imitation Learningcs.LG

Unified framework for imitating tasks across domains with discrepancies.

problem Learning tasks across domains with embodiment, viewpoint, and dynamics mismatches.

method Two-step approach: alignment followed by adaptation. Alignment uses Generative Adversarial MDP Alignment (GAMA) for state and action correspondences from unpaired, unaligned demonstrations. Adaptation leverages these correspondences for zero-shot imitation.

result Effectiveness of the proposed approach in embodiment, viewpoint, and dynamics mismatch scenarios.

Domain Adaptive Imitation Learning MDP Alignment Unsupervised Learning Zero-Shot Imitation Embodiment Mismatch

2019-09-30abs ↗pdf ↗

Provable Offline Reinforcement Learning for Structured Cyclic MDPsstat.ML

CycleFQI tackles offline reinforcement learning for cyclic MDPs, mitigating state distribution mismatch.

problem Offline reinforcement learning for cyclic MDPs with heterogeneous dynamics and discount factors.

method CycleFQI decomposes the cyclic process into stage-wise sub-problems, using vector of stage-specific Q-functions.

result CycleFQI mitigates the curse of dimensionality and provides finite-sample suboptimality error bounds.

reinforcement learning cyclic MDPs offline learning Q-functions state distribution mismatch

2026-02-12abs ↗pdf ↗

Observational Overfitting in Reinforcement Learningcs.LG

Overfitting occurs when RL agents correlate rewards with spurious observation features.

problem Overfitting in reinforcement learning due to correlation with spurious observation features.

method Developed a framework to analyze and design synthetic benchmarks from modified observation spaces.

result Agents can overfit to different observation spaces even if the MDP dynamics are fixed.

reinforcement learning overfitting spurious features Markov Decision Process synthetic benchmarks

2019-12-06abs ↗pdf ↗

Max-Plus Matching Pursuit for Deterministic Markov Decision Processescs.LG

We consider deterministic Markov decision processes (MDPs) and apply max-plus algebra tools to approximate the value iteration algorithm by a smaller-dimensional iteration based on a representation on dictionaries of value functions. The setup naturally leads to novel theoretical results which are simply formulated due…

2019-06-20abs ↗pdf ↗

Multi-level meta-reinforcement learning with skill-based curriculumcs.LG

A new method for solving complex sequential decision-making problems by decomposing them into multiple levels.

problem Sequential decision-making with natural multi-level structure.

method Multi-level meta-reinforcement learning with skill-based curriculum.

result Efficiently reduces stochasticity and policy search space, leading to fewer iterations and computations.

meta-reinforcement learning sequential decision making multi-level structure curriculum learning transferability

2026-03-09abs ↗pdf ↗

Learning Robust State Abstractions for Hidden-Parameter Block MDPscs.LG

This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.

problem Limited observability of state in HiP-MDPs for real-world scenarios with rich observation spaces.

method Inspired by Block MDPs, the work extends HiP-MDPs to enable robust state abstractions for multi-task and meta-reinforcement learning.

result Transfer and generalization bounds based on task and state similarity, and sample complexity bounds that depend on the aggregate number of samples across tasks.

Robust State Abstractions Hidden-Parameter Markov Decision Processes Block Markov Decision Processes Multi-task Reinforcement Learning Meta-reinforcement Learning

2020-07-14abs ↗pdf ↗

Bayesian learning of the optimal action-value function in a Markov decision processstat.ML

Bayesian framework for learning optimal action-value function in MDPs.

problem Uncertainty quantification in MDPs for optimal decision-making strategies.

method Full Bayesian framework including modelling, inference, and decision-making.

result Demonstrates exploration benefits of posterior sampling in MDPs.

Bayesian learning Markov decision process action-value function Monte Carlo inference Thompson sampling

2025-05-03abs ↗pdf ↗

Low-Rank MDPs with Continuous Action Spacescs.LG

New method extends low-rank MDPs to continuous action spaces.

problem Limited applicability of current low-rank MDP methods to continuous action spaces.

method Extending FLAMBE algorithm to continuous action spaces with Hölder smoothness conditions.

result Similar PAC bound achieved for continuous actions with polynomial dependence on smoothness order.

reinforcement learning Markov Decision Processes continuous actions Hölder smoothness PAC bounds

2023-11-06abs ↗pdf ↗

A Model Selection Approach for Corruption Robust Reinforcement Learningcs.LG

New model selects robustly in adversarial reinforcement learning with unknown corruption.

problem Adversarial corruption in reinforcement learning with unknown total corruption amount.

method Model selection approach for finite-horizon tabular and linear MDPs.

result First worst-case optimal bound without knowledge of total corruption.

reinforcement learning corruption robustness model selection adversarial tabular MDPs

2021-10-07abs ↗pdf ↗

Relax but stay in control: from value to algorithms for online Markov decision processescs.LG

Online learning algorithms are designed to perform in non-stationary environments, but generally there is no notion of a dynamic state to model constraints on current and future actions as a function of past actions. State-based models are common in stochastic control settings, but commonly used frameworks such as Mark…

2013-10-28abs ↗pdf ↗

Modular Deep Reinforcement Learning with Temporal Logic Specificationscs.LG

A reinforcement learning framework for Mars rover control using temporal logic.

problem Sparse rewards in continuous-state continuous-action MDPs with high-level temporal structures.

method Actor-critic, model-free, online RL framework with modular DDPG architecture.

result Success rate of synthesised policy in Mars rover experiment.

Reinforcement Learning Temporal Logic Markov Decision Processes Deep Deterministic Policy Gradient Mars Rover

2019-09-23abs ↗pdf ↗

Deep neural networks can provably solve Bellman equations for Markov decision processes without the curse of dimensionalitymath.OC

Deep nets solve MDPs without high dimensions.

problem Solving Bellman equations for MDPs in high dimensions.

method Deep neural networks with ReLU activation approximating payoff and transition functions.

result Deep nets can approximate

Q

-functions in polynomially bounded parameters.

Deep learning Reinforcement learning Markov decision processes Bellman equations Optimal control

2025-06-28abs ↗pdf ↗

A Unifying Framework for Reinforcement Learning and Planningcs.LG

This paper presents a unifying framework for reinforcement learning and planning.

problem Sequential decision making in AI, formalized as MDP optimization.

method A unifying algorithmic framework (FRAP) for reinforcement learning and planning.

result Identifies common dimensions in MDP planning and learning algorithms.

Reinforcement Learning Planning Markov Decision Process Artificial Intelligence Algorithmic Design

2020-06-26abs ↗pdf ↗

Reinforcement Learning with Action-Triggered Observationscs.LG

New framework for reinforcement learning with sporadic state observations.

problem Partial observability in reinforcement learning.

method Action-Triggered Sporadically Traceable Markov Decision Processes (ATST-MDPs).

result Optimistic algorithm achieving regret bound for episodic learning.

Reinforcement Learning Partial Observability Sporadic Observations Markov Decision Processes Optimistic Algorithms

2025-10-02abs ↗pdf ↗

Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPscs.LG

Adaptive reduction scheme approximates optimal policy in regularized MDPs.

problem Finding near optimal policy in regularized MDPs with biased solutions.

method Adaptive reduction of regularization parameter λ to approximate optimal policy.

result Iteration complexity reduced for obtaining ε-optimal policy.

MDPs Regularization Policy approximation Adaptive reduction Optimization

2020-10-31abs ↗pdf ↗

Sample Complexity Bounds for Linear Constrained MDPs with a Generative Modelcs.LG

New algorithm solves constrained MDPs with generative model, reducing sample complexity.

problem Maximizing rewards while adhering to constraints in MDPs.

method Primal-dual framework using mirror descent value iteration.

result Near-optimal sample complexity bounds for CMDPs.

Markov decision processes constrained optimization sample complexity generative model mirror descent

2025-07-02abs ↗pdf ↗

Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learningstat.ML

We study Exo-MDPs to reduce sample complexity in reinforcement learning.

problem Reducing sample complexity in reinforcement learning for structured MDPs.

method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.

result Proved

O(H^{3/2}d\sqrt{K})

regret bound for Exo-MDPs, matching lower bounds.

Reinforcement Learning Markov Decision Processes Exogenous States Sample-Efficient Inventory Control

2024-09-22abs ↗pdf ↗

Invariant Causal Prediction for Block MDPscs.LG

Proposes method to learn state abstractions that generalize across environments.

problem Learning abstractions that generalize in block MDPs.

method Invariant causal prediction to learn model-irrelevant state abstractions (MISA).

result Proves high probability of outputting a state abstraction corresponding to causal feature set for return.

reinforcement learning block MDPs causal inference state abstractions generalization

2020-03-12abs ↗pdf ↗

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methodsmath.OC

Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.

problem Optimal policies in finite-time MDPs are not stationary and require epoch-specific training.

method Introduces dynamic policy gradient combining dynamic programming and policy gradient, analyzes convergence for softmax parametrisation.

result Dynamic policy gradient training exploits finite-time structure, leading to better convergence bounds.

policy gradient dynamic programming finite horizon MDPs softmax parametrisation convergence analysis

2023-10-04abs ↗pdf ↗