New algorithm solves uncertain Markov decision processes using Wasserstein uncertainty.
problem Solving Markov decision processes with uncertain transition probabilities.
method Distributionally robust Q Q Q -learning algorithm for Wasserstein uncertainty. result Convergence of the algorithm proved and demonstrated with real data.
We develop robust Markov Decision Processes with risk measures for uncertain environments.
problem Uncertainty in Markov Decision Processes and its impact on risk measures.
method Formulation as a Stackelberg game, robust cost and value iterations, existence of optimal policies.
result Existence of deterministic optimal policies for robust optimization and risk measures.
This paper analyzes risk-sensitive reinforcement learning with Conditional Value-at-Risk (CVaR) for robust Markov Decision Processes.
problem Risk-sensitive reinforcement learning for robust Markov Decision Processes (RMDPs) with state-action-dependent ambiguity sets.
method The paper establishes a connection between robustness and risk sensitivity, defining a new risk measure NCVaR and proposing value iteration algorithms.
result The proposed approach using NCVaR optimization and value iteration algorithms can solve problems with state-action-dependent ambiguity sets.
The paper tackles robust policy learning in MDPs using statistical methods.
problem Offline data-driven sequential decision making in MDPs.
method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.
result Established a rate-optimal regret bound up to a logarithmic factor.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
A method to learn robust policies for environments with model mismatches.
problem Training agents in high-stakes scenarios with mismatched training and real environments.
method Formalizes the perturbation as a zero-sum game to find Nash Equilibrium, which corresponds to the robust policy.
result Our algorithm can find a near-optimal robust policy with high probability using polynomial samples.
Bayesian regularization improves policy performance in noisy MDPs.
problem Suboptimal policies from estimated model parameters.
method Bayesian regularization of MDP objective function with prior information.
result Regularized policies show better robustness against model noise.
DR-RPO optimizes robust policies in RL with limited interaction, achieving sublinear regret.
problem Policy optimization in RL under distribution shift and adversarial dynamics.
method DR-RPO algorithm incorporating reference-policy regularization and upper confidence bonus for exploration.
result DR-RPO achieves sublinear regret and polynomial suboptimality bounds in robust RL.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
New method removes oracle and reduces memory usage for robust MDPs.
problem Applying robust MDPs in practice due to model estimation and oracle requirements.
method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.
result Sample-efficient algorithm with lower storage requirement and no oracle.
New framework for robust reinforcement learning policies in uncertain environments.
problem Robust reinforcement learning policies in environments with distributional shifts.
method Comprehensive modeling framework centered around robust Markov decision processes (RMDPs).
result Existence and conditions for the dynamic programming principle (DPP) in RMDPs.
New method for robust policy evaluation in offline reinforcement learning with sequentially exogenous unobserved confounders.
problem Offline reinforcement learning in domains with unobserved confounders.
method Orthogonalized robust fitted-Q-iteration with closed-form solutions and bias-correction.
result Effective in simulations and real-world data, improving robustness and computational ease.
We introduce a new formulation of the Hidden Parameter Markov Decision Process (HiP-MDP), a framework for modeling families of related tasks using low-dimensional latent embeddings. Our new framework correctly models the joint uncertainty in the latent parameters and the state space. We also replace the original Gaussi…
A new framework for robust policy learning in MDPs with linear mixture dynamics.
problem Off-dynamics challenge in real-world decision-making problems.
method Linear mixture DRMDP framework, meta algorithm for robust policy learning.
result The new framework provides a more refined representation of uncertainties.
New algorithms improve robust reinforcement learning under uncertainty.
problem Robust reinforcement learning in MDPs with contamination.
method Non-asymptotic convergence analysis of Q Q Q -learning and actor-critic methods. result Efficient algorithms learn robust policies with minimal samples.
Framework for optimizing portfolios under model uncertainty.
problem Optimizing portfolios in volatile markets considering model uncertainty.
method Dynamic programming and robust optimization for Markov decision processes.
result Robust optimization leads to better portfolio strategies in uncertain market conditions.
The paper tackles robust reinforcement learning with performance guarantees.
problem Finding a robust policy for RMDP with state space uncertainties.
method Proposes RLSPI algorithm for learning optimal robust policy with performance bounds.
result Demonstrates the performance of RLSPI on standard benchmark problems.
This work extends HiP-MDPs to robust state abstractions for multi-task and meta-reinforcement learning.
problem Limited observability of state in HiP-MDPs for real-world scenarios with rich observation spaces.
method Inspired by Block MDPs, the work extends HiP-MDPs to enable robust state abstractions for multi-task and meta-reinforcement learning.
result Transfer and generalization bounds based on task and state similarity, and sample complexity bounds that depend on the aggregate number of samples across tasks.
A method for robust reinforcement learning in large state spaces.
problem Challenges in RL with large state spaces, costly data, and real-world dynamics deviation.
method Distributionally robust Markov decision processes with Gaussian Processes and maximum variance reduction.
result Efficient learning of multi-output nominal transition dynamics with statistical sample complexity bounds.
A new algorithm trains experts to safely guide agents in partially observed environments.
problem Existing imitation learning methods for POMDPs can lead to sub-optimal or unsafe policies.
method Derive an objective to encourage the expert to maximize the agent's reward, then use it to train both expert and agent.
result The algorithm produces an expert policy that the agent can safely imitate, outperforming fixed expert policies.
New method reduces sample complexity for robust reinforcement learning.
problem Finite sample analysis in robust reinforcement learning.
method Stochastic approximation framework with controlled bias, using MLMC techniques and geometric truncation.
result Order-optimal sample complexity of i l d e O ( ε − 2 ) ilde{\mathcal{O}}(ε^{-2}) i l d e O ( ε − 2 ) for robust policy evaluation. We develop a robust RL algorithm for off-dynamics environments with improved suboptimality bounds and computational efficiency.
problem Learning policies robust to uncertainties in transition dynamics between training and deployment environments.
method Distributionally robust Markov decision processes (DRMDPs) with a novel algorithm We-DRIVE-U.
result Improved suboptimality bound of O ~ ( d H ⋅ min { 1 / ρ , H } / K ) \widetilde{\mathcal{O}}\big({d H \cdot \min \{1/ρ, H\}/\sqrt{K} }\big) O ( d H ⋅ min { 1/ ρ , H } / K ) , near-optimal up to O ( H ) \mathcal{O}(\sqrt{H}) O ( H ) . Improved reinforcement learning for environments with distributional shifts.
problem Learning optimal policies in environments with distributional shifts.
method Distributionally robust Q-learning with multi-level Monte Carlo estimator.
result Proved upper bound on sample complexity for robust RL.
The paper studies risk-sensitive MDPs with recursive risk measures.
problem Risk-sensitive decision-making in MDPs with unbounded costs.
method Recursive application of static risk measures, Bellman equation derivation, existence of optimal policies.
result Existence of Markovian optimal policies for infinite planning horizons, contractive model for stationary optimal policy.
PDTS improves robustness in sequential decision-making.
problem Robust active task sampling for efficient and reliable decision-making.
method Characterizes robust active task sampling as a Markov decision process, proposes PDTS method.
result Significantly improves zero-shot and few-shot adaptation robustness.
A new algorithm for deep Q-learning with robustness to state transition uncertainty.
problem Model uncertainty in state transitions for non-tabular, continuous state spaces.
method Distributionally robust approach using worst-case transition ball and dualized Bellman operator with Sinkhorn distance.
result Optimal policy found through solving non-linear Bellman equation with neural network parameterization.
New algorithms learn robust policies from shifted distributions.
problem Learning robust policies in environments with distributional shifts.
method Two novel model-free algorithms: distributionally robust Q-learning and variance-reduced distributionally robust Q-learning.
result Achieves minimax sample complexity upper bound of i l d e O ( ∣ S ∣ ∣ A ∣ ( 1 − γ ) − 4 ε − 2 ) ilde O(|\mathbf{S}||\mathbf{A}|(1-γ)^{-4}ε^{-2}) i l d e O ( ∣ S ∣∣ A ∣ ( 1 − γ ) − 4 ε − 2 ) . Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. We consider for the first time the semiparametric efficiency limits of OPE in Markov decision processes (MDPs), where actions, rewar…
Improved off-policy evaluation for MDPs with weak distributional overlap.
problem Evaluation of policies when target and data-collection distributions are not strongly overlapping.
method Truncated Doubly Robust (TDR) estimators for off-policy evaluation in MDPs under weak distributional overlap.
result TDR estimators can recover large-sample behavior and are consistent even when distribution ratios are not square-integrable.
Paper studies robust MDPs, improving sample complexity and asymptotic performance.
problem Optimal robust policy and value function in robust MDPs with generative models.
method Improves prior results on non-asymptotic and asymptotic performances of robust MDPs, considering various uncertainty sets.
result Improved sample complexity and asymptotic normality of optimal robust value function.
Paper develops a new estimator for MDPs' risk functionals with lower variance and bias.
problem Estimating the distribution of returns in MDPs with high variance and bias.
method Developed a doubly robust (DR) estimator for the CDF of returns in MDPs, incorporating model-based estimation to mitigate variance issues.
result The DR estimator achieves lower variance and bias compared to IS estimators, and matches minimax lower bounds.
This paper solves the open problem of computing Bayes optimal prediction for decision trees using a Markov chain Monte Carlo method.
problem Computing the Bayes optimal prediction for decision trees is infeasible due to an infeasible summation over all division patterns of a feature space.
method Solved the open problem using a Markov chain Monte Carlo method with adaptively tuned step size.
result Computed the Bayes optimal prediction for decision trees using a Markov chain Monte Carlo method.
We develop efficient and sharp bounds on policy value under perturbations in MDPs.
problem Evaluating policies under best- and worst-case perturbations in MDPs with transition observations.
method Proposed a perturbation model for MDPs, developed semiparametrically efficient estimator with asymptotic normality.
result Semiparametrically efficient and asymptotically normal estimator for policy value bounds.
Paper tackles robust offline RL for non-Markovian processes, improving efficiency and applicability.
problem Learning robust policies for non-Markovian decision processes with limited offline data.
method Proposes a novel algorithm with dataset distillation and LCB design for robust values, derived new dual forms, and introduces concentrability coefficients.
result Proves polynomial sample efficiency for finding ε-optimal robust policies.
Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.
Paper tackles robust reinforcement learning with minimal data.
problem Learning robust policies from limited data in uncertain environments.
method Distributionally robust formulation, model-based algorithm combining value iteration and pessimism.
result Proves near-optimal sample complexity for robust offline RL.
Paper presents an algorithm for optimal regret in communicating Markov decision processes.
problem Achieving optimal regret in Markov decision processes with a communicating assumption.
method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.
result The algorithm achieves asymptotically optimal regret K ( M ) log ( T ) + o ( log ( T ) ) K(M) \log(T) + \mathrm{o}(\log(T)) K ( M ) log ( T ) + o ( log ( T )) for communicating Markov decision processes. A new MDP with Bandits approach for sequential decision making in linear-flow scenarios.
problem Sequential decision making with limited feedback in a linear-flow context.
method Formulated as an MDP with Bandits, using Thompson sampling for action selection and exact dynamic programming for allocation.
result The proposed MDP with Bandits algorithm outperforms other methods in sequential decision making.
Paper proposes DRL for unsupervised IoT localization.
problem Challenges in unsupervised localization of IoT devices.
method Modeling localization as MDP, using DRL with reward-setting and RSS measurements.
result Demonstrated effectiveness of DRL in wireless localization.
Paper develops IV method for consistent OPE in confounded MDPs.
problem Confounding variables in observational data affect OPE effectiveness.
method Instrumental variable approach for consistent off-policy evaluation.
result IV method enables correct identification of target policy's value.
Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.
problem Optimality in safety-constrained Markov decision processes with multichain structure.
method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.
result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.
A new meta-learner improves prediction of individualized outcomes in sequential decisions.
problem Predicting individualized outcomes over long horizons in sequential decision-making.
method Developed a novel meta-learner called DRQ-learner with theoretical guarantees of orthogonality and quasi-oracle efficiency.
result DRQ-learner achieves quasi-oracle efficiency, doubly robustness, and Neyman-orthogonality.
Advances in mobile computing technologies have made it possible to monitor and apply data-driven interventions across complex systems in real time. Markov decision processes (MDPs) are the primary model for sequential decision problems with a large or indefinite time horizon. Choosing a representation of the underlying…
A Robust Markov Decision Process (RMDP) is a sequential decision making model that accounts for uncertainty in the parameters of dynamic systems. This uncertainty introduces difficulties in learning an optimal policy, especially for environments with large state spaces. We propose two algorithms, RTD-DQN and Deep-RoK, …
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an…
Develops an actor-critic algorithm for risk-sensitive Markov decision processes.
problem Risk-sensitive cost criterion in Markov decision processes.
method Actor-critic algorithm with function approximation.
result Asymptotic convergence of the actor-critic algorithm.
Overview of risk-sensitive Markov decision processes with Optimized Certainty Equivalent.
problem Optimizing decision-making under risk in Markov processes.
method Analyzes risk-sensitive criteria using Optimized Certainty Equivalent, including entropic risk and Conditional Value-at-Risk.
result Conditions for the existence of optimal policies and solution procedures are provided.