Optimal policies in Markov decision processes (MDPs) are very sensitive to model misspecification. This raises serious concerns about deploying them in high-stake domains. Robust MDPs (RMDP) provide a promising framework to mitigate vulnerabilities by computing policies with worst-case guarantees in reinforcement learn…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “MDP Ambiguity”
We address the problem of computing reliable policies in reinforcement learning problems with limited data. In particular, we compute policies that achieve good returns with high confidence when deployed. This objective, known as the \emph{percentile criterion}, can be optimized using Robust MDPs~(RMDPs). RMDPs general…
Robust MDPs (RMDPs) can be used to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution are determined by the ambiguity set---the set of plausible transition probabilities---which is usually constructed as a multi-dimensional confidence region. E…
Robustness is important for sequential decision making in a stochastic dynamic environment with uncertain probabilistic parameters. We address the problem of using robust MDPs (RMDPs) to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution is det…
Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.
problem Non-rectangular robust Markov decision processes under average-reward criterion.
method Proves history-dependent policies are robust-optimal, introduces transient-value framework, constructs epoch-based policy.
result Existence and properties of robust optimal policies, transient value bounds.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
Meta-agent learns effective exploration from offline data.
problem Design a meta-agent to quickly maximize reward in unseen tasks.
method Bayesian RL approach with adaptive neural belief estimate.
result Meta-agent learns effective exploration behavior from diverse tasks.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
This paper analyzes risk-sensitive reinforcement learning with Conditional Value-at-Risk (CVaR) for robust Markov Decision Processes.
problem Risk-sensitive reinforcement learning for robust Markov Decision Processes (RMDPs) with state-action-dependent ambiguity sets.
method The paper establishes a connection between robustness and risk sensitivity, defining a new risk measure NCVaR and proposing value iteration algorithms.
result The proposed approach using NCVaR optimization and value iteration algorithms can solve problems with state-action-dependent ambiguity sets.
Higher-Order Ambiguity Attitudesq-fin.RM
Model-free preference under ambiguity defined and applied.
problem Understanding and quantifying ambiguity aversion and prudence.
method Introduces a new model-free definition of ambiguity attitudes and applies it in various contexts.
result New definition of ambiguity prudence equivalent to specific mathematical functions.
Investment strategy optimized for ambiguity and interest rate risk.
problem Dynamic asset allocation with interest rate risk and ambiguity.
method Closed-form solution for optimal investment strategy.
result Ambiguity affects speculative motives, not hedging of interest rate risk.
Study nonconcave portfolio choice with smooth ambiguity and Bayesian learning.
problem Nonconcave portfolio choice under smooth ambiguity and Bayesian learning.
method Developed a general framework for dynamic, non-concave asset allocation.
result Dynamic consistency achieved through a robust representation.
In this paper, we study optimal switching problems under ambiguity. To characterize the optimal switching under ambiguity in the finite horizon, we use multidimensional reflected backward stochastic differential equations (multidimensional RBSDEs) and show that a value function of the optimal switching under ambiguity …
Study insurance pricing under correlation ambiguity without increasing prices or reducing utility.
problem Understanding the dependence structure between insurance and financial risks.
method Dynamic equilibrium analysis of insurance pricing with worst-case beliefs.
result Correlation ambiguity does not necessarily increase insurance prices or reduce insurers' utility.
New formulations capture aversion to ambiguity about volatility.
problem Capturing aversion to ambiguity about unknown and time-varying volatility.
method Introduces novel preference formulations and compares them with existing models.
result Illustrates the impact of ambiguity aversion in static and dynamic models.
A framework for robust exploration in reinforcement learning under ambiguity.
problem Optimal stopping under ambiguity in reinforcement learning.
method Continuous-time robust reinforcement learning framework using -expectation and backward stochastic differential equations.
result Constructs a robust exploratory stopping time approximating the optimal stopping time under ambiguity.
Study inert and ambiguous classes in modular group using combinatorial methods.
problem Counting inert and ambiguous conjugacy classes in modular group.
method Purely combinatorial approach using word length in free product representation.
result Exact counting formulas and asymptotic growth rates for inert and ambiguous classes.
Investment strategy in ambiguous financial markets with learning
problem Continuous time investment problem in multi-asset Black-Scholes market with model ambiguity
method Optimal dynamic investment strategy within the class of all adapted strategies which allow for learning
result Ambiguity averse investors invest less in risky assets
Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
We study the dynamic indifference pricing with ambiguity preferences. For this, we introduce the dynamic expected utility with ambiguity via the nonlinear expectation--G-expectation, introduced by Peng (2007). We also study the risk aversion and certainty equivalent for the agents with ambiguity. We obtain the dynamic …
Model cash management under ambiguity using maxmin preferences and diffusion.
problem Optimizing cash reserves in the presence of ambiguity.
method Singular control model with maxmin preferences, verified using Dynkin games.
result Higher expected costs and narrower inaction region under increased ambiguity.
We study Exo-MDPs to reduce sample complexity in reinforcement learning.
problem Reducing sample complexity in reinforcement learning for structured MDPs.
method Introducing Exo-MDPs and proving structural equivalence to linear mixture MDPs, establishing regret bounds.
result Proved regret bound for Exo-MDPs, matching lower bounds.
In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyz…
Paper investigates Lambda Value-at-Risk under ambiguity and risk sharing.
problem Investigates Lambda Value-at-Risk under ambiguity and risk sharing.
method Establishes equivalence of robust VaR and traditional VaR under ambiguity sets, analyzes properties, derives explicit formulas, and explores risk sharing.
result Unified and extended the concept of Value-at-Risk under ambiguity, derived explicit formulas for specific ambiguity sets, and explored risk sharing.
The paper explores continuous inverse ambiguous functions on various Lie groups.
problem Existence of continuous inverse ambiguous functions on Lie groups.
method Investigation of continuous inverse ambiguous functions on specific Lie groups.
result Existence of continuous inverse ambiguous functions on various Lie groups.
Paper establishes new lower bounds for MDPs with changing transition kernels.
problem Minimizing sample complexity and regret in non-stationary MDPs.
method Developed novel lower bounds and constructed hard MDPs.
result Proved sample complexity lower bound.
An unconventional approach for optimal stopping under model ambiguity is introduced. Besides ambiguity itself, we take into account how ambiguity-averse an agent is. This inclusion of ambiguity attitude, via an -maxmin nonlinear expectation, renders the stopping problem time-inconsistent. We look for subgame perfect…
We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present …
Improves DRO with Bayesian Ambiguity Sets for model misspecification.
problem Overly conservative decisions due to misspecified models in DRO.
method Introduces DRO-RoBAS with robust posterior predictive distribution.
result Outperforms other Bayesian and empirical DRO approaches in out-of-sample performance.
Study optimal timing to divest from assets with uncertain future scenarios.
problem Optimal timing to divest from assets with uncertain future scenarios.
method Smooth model of decision making under ambiguity aversion, optimal stopping problem with learning.
result Proves a minimax result reducing the problem to standard optimal stopping problems with learning.
New method approximates POMDPs with PB-MDPs, providing error bounds and practical algorithms.
problem Difficulty in solving POMDPs with continuous or hybrid state and observation spaces.
method Bounding particle filtering error and adapting MDP algorithms to POMDPs.
result General theory and practical algorithms for POMDPs with no direct dependence on state and observation space sizes.
This paper compares average-K and top-K classification methods under ambiguity.
problem Choosing a single label in ambiguous cases leads to low precision.
method Formally characterizes ambiguity profiles and compares average-K and top-K classification methods.
result Average-K can achieve lower error rates than top-K in some ambiguous cases.
Proposes handling ambiguity in sequential data predictions.
problem Handling uncertainty in sequential data predictions.
method Extension of MHP model to recurrent architectures, introducing a novel metric.
result Achieved promising results on various sequential data tasks.
This paper compares different DRO formulations for pension fund management.
problem Navigating uncertainty in asset liability management for pension funds.
method Three DRO formulations: mixture, box, and Wasserstein ambiguity sets.
result Wasserstein and box ambiguity sets outperform traditional approaches in fund performance.
We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimen…
Optimizes learning policies in MDPs with weakly communicating structure.
problem Learning optimal policies in weakly communicating MDPs with generative model.
method Span-based approach, reducing to discounted MDPs for analysis.
result First minimax optimal sample complexity bound for weakly communicating MDPs.
Adapts AUM to identify ambiguous tasks in crowdsourced learning, improving generalization.
problem Discerning ambiguous tasks in crowdsourced labels to prevent mislabeling.
method Introduces Weighted Areas Under the Margin (WAUM) to average AUMs weighted by task-specific scores.
result Improves generalization performance by discarding ambiguous tasks.
New risk measures for quantiles under ambiguity improve risk sharing.
problem Risk optimization under ambiguity using quantiles.
method Introducing Choquet quantiles and Choquet Expected Shortfall.
result Optimal allocations for quantile agents under ambiguity.
A Markov Decision Process (MDP) is a popular model for reinforcement learning. However, its commonly used assumption of stationary dynamics and rewards is too stringent and fails to hold in adversarial, nonstationary, or multi-agent problems. We study an episodic setting where the parameters of an MDP can differ across…
New method removes oracle and reduces memory usage for robust MDPs.
problem Applying robust MDPs in practice due to model estimation and oracle requirements.
method Transformed robust MDPs into an alternative form allowing stochastic gradient methods and model-free approach.
result Sample-efficient algorithm with lower storage requirement and no oracle.
DeepAveragers solves offline RL by solving derived MDPs from static data.
problem Offline reinforcement learning with limited data.
method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.
result The approach can lower-bound performance and scale to complex offline RL problems.
Online RL in Linearly -Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignorecs.LG
New RL method learns to skip states in linearly -realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly -realizable action-values.
method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly -realizable MDPs.
Optimizes learning policies in average-reward MDPs with improved sample complexity.
problem Learning optimal policies in average-reward MDPs with limited samples.
method Reduces to discounted MDPs and uses improved bounds for variance parameters.
result Establishes minimax optimal sample complexity bound of O(SA(H/ε^2))
Researchers resolved ambiguities in gravitational radiation charges.
problem Ambiguities in charges related to gravitational radiation.
method Addressed supertranslation ambiguities in classical and extended BMS algebras.
result Proposed and proved an invariant charge free from supertranslation ambiguity.
Paper tackles robust control of SDEs with ambiguity, proving value function existence and applying to investment problems.
problem Robust control of SDEs with ambiguity parameters and non-Lipschitz coefficients.
method Existence and uniqueness of value function established through BSDEs with non-linear growth conditions.
result Existence and uniqueness of value function in proper space, verified through BSDEs.
According to conventional wisdom, ambiguity accelerates optimal timing by decreasing the value of waiting in comparison with the unambiguous benchmark case. We study this mechanism in a multidimensional setting and show that in a multifactor model ambiguity does not only influence the rate at which the underlying proce…
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
We solve POMDPs by approximating them as finite-state MDPs.
problem Computational challenges in learning optimal policies for POMDPs.
method Transform POMDP into a Superstate MDP, apply TD-learning and policy optimization.
result Finite-time bounds on TD-learning error for non-Markovian dynamics.