Deep RL policies are vulnerable to adversarial perturbations, but vanilla training yields more robust policies.
problem Vulnerability of deep reinforcement learning policies to adversarial perturbations.
method Analysis of deep reinforcement learning policy landscape and comparison of vanilla vs. adversarial training.
result Vanilla training yields more robust policies compared to adversarial training.
The paper tackles robust reinforcement learning with performance guarantees.
problem Finding a robust policy for RMDP with state space uncertainties.
method Proposes RLSPI algorithm for learning optimal robust policy with performance bounds.
result Demonstrates the performance of RLSPI on standard benchmark problems.
Improves reinforcement learning policies for robustness.
problem Lack of robustness in reinforcement learning policies.
method Risk-aware Distributional Reinforcement Learning (SDPG) with CVaR.
result Risk-averse policies achieve robustness against disturbances.
We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …
Framework learns robust control policies from expert demonstrations.
problem Adversarial robustness and closed-loop generalization in feedback control policies.
method Lipschitz-constrained loss minimization for certified robustness and generalization.
result Finite sample bound on policy learning error and robust closed-loop stability.
The paper tackles robust policy learning in MDPs using statistical methods.
problem Offline data-driven sequential decision making in MDPs.
method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.
result Established a rate-optimal regret bound up to a logarithmic factor.
Social Security and other public policies can be viewed as a series of cash in and outflows that depend on parameters such as the age distribution of the population and the retirement age. Given forecasts of these parameters, policies can be designed to be financially stable, i.e., to terminate with a zero balance. If …
DR-RPO optimizes robust policies in RL with limited interaction, achieving sublinear regret.
problem Policy optimization in RL under distribution shift and adversarial dynamics.
method DR-RPO algorithm incorporating reference-policy regularization and upper confidence bonus for exploration.
result DR-RPO achieves sublinear regret and polynomial suboptimality bounds in robust RL.
A new method learns robust policies from offline data with latent structures.
problem Conservative policies under unrealistic dynamics shifts.
method d-RRMDP framework with f-divergence regularization and R2PVI algorithm. result R2PVI learns robust policies with superior computational efficiency.
RRPI improves offline RL by optimizing policies against worst-case dynamics.
problem Offline RL's performance degrades under distribution shift and transition uncertainty.
method Formulates offline RL as robust policy optimization, treating transition kernel as decision variable.
result RRPI achieves strong average performance on D4RL benchmarks, outperforming recent baselines.
AdaRL improves robust RL by adaptively adjusting policy complexity.
problem Handling epistemic uncertainty in environment dynamics.
method Bi-level optimization framework with adaptive rank adjustment.
result AdaRL outperforms existing methods on MuJoCo benchmarks.
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.
This paper tackles robust policy learning under concept drifts, improving upon existing methods.
problem Tackles robust policy learning under concept drifts, improving upon existing methods.
method Develops a doubly-robust estimator and a learning algorithm to maximize policy value within a given policy class.
result The proposed algorithm achieves sub-optimality gap of the order κ(Π)n−1/2, demonstrating substantial improvement over existing benchmarks. New framework improves sample efficiency and robustness in RL with smooth policies.
problem Sample inefficiency and lack of robustness in deep reinforcement learning.
method SR^2L framework, smoothness-inducing regularization.
result Improved sample efficiency and robustness in both on-policy and off-policy RL algorithms.
Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally …
This work proposes robust reinforcement learning methods using both offline and online data.
problem Designing robust policies against parameter uncertainties in high-dimensional systems.
method Proposes RPQ for model-free learning with historical data and HyTQ for hybrid learning with both historical and online data.
result Unified analysis and theoretical guarantees for robust optimal policies in high-dimensional systems.
Bayesian investor learns unknown asset drift, trades mean-variance optimal portfolio, but policy is robust to observation model distortion.
problem Bayesian portfolio selection with observation model distortion
method Robust Bayesian portfolio selection
result Robust policy and its price are closed form, with price of robustness half the variance of the non-robust investor's loss.
Study non-rectangular robust MDPs for average-reward, finding optimal policies and transient values.
problem Non-rectangular robust Markov decision processes under average-reward criterion.
method Proves history-dependent policies are robust-optimal, introduces transient-value framework, constructs epoch-based policy.
result Existence and properties of robust optimal policies, transient value bounds.
New RL algorithm learns robust policies without knowing nominal model.
problem Designing robust policies for RL with unknown nominal model.
method Model-based RL algorithm with three uncertainty set forms.
result Precise sample complexity for each uncertainty set.
The paper tackles robust policy learning from multiple data sources.
problem Learning a policy that generalizes across diverse settings from multiple heterogeneous data sources.
method Proposes a minimax regret optimization objective and a policy learning algorithm combining doubly robust offline policy evaluation and no-regret learning.
result Achieves minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources.
Paper tackles robust policy learning with incomplete data.
problem Learning policies from past data assumes future environment is the same, which is often false.
method Develops a distributionally robust policy evaluation and learning algorithm.
result Proposed algorithm provides robustness to adversarial perturbations and covariate shifts.
The paper shows how policy regularization acts like an adversary to improve robustness.
problem Improving robustness of learned policies in reinforcement learning.
method Using convex duality, the paper characterizes adversarial reward perturbations and provides generalization guarantees.
result Policy regularization acts as an adversary to improve robustness against worst-case reward perturbations.
SCPO learns robust policies without modeling disturbance, improving real-world task performance.
problem Poor performance of reinforcement learning in real-world tasks due to disturbance in transition dynamics.
method State-conservative policy optimization (SCPO) that reduces disturbance to state space and approximates it with a gradient-based regularizer.
result SCPO learns robust policies without prior knowledge of disturbance or simulators, improving performance in robot control tasks.
A method to learn robust policies for environments with model mismatches.
problem Training agents in high-stakes scenarios with mismatched training and real environments.
method Formalizes the perturbation as a zero-sum game to find Nash Equilibrium, which corresponds to the robust policy.
result Our algorithm can find a near-optimal robust policy with high probability using polynomial samples.
Proposes RFQI for robust RL using offline data.
problem Learning robust policies in the presence of model uncertainty.
method RFQI algorithm using offline data to learn optimal robust policy.
result RFQI learns near-optimal robust policy under standard assumptions.
This paper investigates the resilience and robustness of Deep Reinforcement Learning (DRL) policies to adversarial perturbations in the state space. We first present an approach for the disentanglement of vulnerabilities caused by representation learning of DRL agents from those that stem from the sensitivity of the DR…
As all physical adaptive quantum-enhanced metrology schemes operate under noisy conditions with only partially understood noise characteristics, so a practical control policy must be robust even for unknown noise. We aim to devise a test to evaluate the robustness of AQEM policies and assess the resource used by the po…
A new policy learning method allows policies to abstain when uncertain, improving safety and applicability.
problem Risk of making decisions without full confidence in uncertain predictions.
method Policy learning with abstention, identifying near-optimal policies and constructing an abstention rule.
result Improved safety and applicability in policy learning, with theoretical guarantees.
Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
Extends double linear policy with time-varying weights and proves robust positive expectation.
problem Ensuring robustness in policy optimization with time-varying parameters.
method Employed a novel elementary symmetric polynomials characterization approach to prove robust positive expectation (RPE). Derived explicit expressions for expected cumulative gain-loss and variance.
result Proved the robust positive expectation property holds for the extended double linear policy.
Regularized policies are robust to adversarial rewards.
problem Understanding the effects of regularization on policy exploration and robustness.
method Using Fenchel duality to derive the dual problem of the regularized RL objective, showing the optimal policy is robust to adversarial rewards.
result Regularized policies are optimal for a reinforcement learning problem under adversarial reward conditions.
Discuss new policy learning objectives and methods.
problem Improving policy learning efficiency and robustness.
method Introducing curvature considerations and calibration data methods.
result Efficient retargeting and distributionally robust policies.
Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative. Our soft-robust framework is an attempt to overcome this issue. In this pa…
Step-DAD improves BED by periodically updating a design policy during experiments.
problem Improving flexibility and robustness in Bayesian experimental design.
method Semi-amortized, policy-based approach that updates a design policy during data collection.
result Consistently superior decision-making and robustness compared to current BED methods.
A new estimator for evaluating policies in unknown environments.
problem Evaluating policies when both logging policy and value function are unknown.
method Doubly-Robust (DR) off-policy evaluation (OPE) estimator, DRUnknown, that estimates both the logging policy and value function.
result DRUnknown achieves the smallest asymptotic variance and is optimal when both models are correctly specified.
Robust OPE framework uses human inputs to improve policy evaluation in changing environments.
problem Inaccurate policy evaluations due to shifts in environment properties.
method Adapts OPE methods to shifts on user-inputted covariates, providing more realistic utility estimates.
result Robust OPE framework yields less pessimistic policy evaluations and captures realistic dataset shifts.
New algorithms solve robust MDPs efficiently, significantly faster than existing methods.
problem Computing robust MDP solutions with uncertainty in transition probabilities is computationally expensive.
method Partial policy iteration and fast robust Bellman operator computation methods.
result The proposed methods are many orders of magnitude faster than state-of-the-art approaches.
CPME embeds counterfactual outcomes in RKHS for flexible policy evaluation.
problem Estimating counterfactual policy outcomes for decision-making.
method Counterfactual Policy Mean Embedding (CPME) framework in RKHS, plug-in and doubly robust estimators, kernel test statistic.
result Doubly robust estimator improves convergence rates and asymptotic normality.
The paper develops a method to learn robust decision policies from observational data, reducing high-cost outcomes.
problem Learning safe decision policies from observational data with high-risk outcomes.
method Develops a method to learn policies that reduce high-cost outcomes, valid under finite samples and uneven feature overlap.
result Validates the method with real and synthetic data, providing statistical bounds on decision costs.
New methods improve robust decision-making under uncertainty in off-policy evaluation.
problem Statistical uncertainty and causal considerations in off-policy evaluation.
method Marginal Ratio (MR) estimator, Conformal Off-Policy Prediction (COPP), causal bounds.
result Improved robustness and uncertainty quantification in off-policy decision-making.
Estimates personalized policies robust to shifts in target populations.
problem Estimating policies that perform well in diverse target populations.
method Develops methods for estimating robust policies considering shifts in outcomes and characteristics.
result Welfare-maximizing policies are robust to certain shifts in potential outcomes.
Paper develops robust policy evaluation for reinforcement learning with outlier and heavy-tailed rewards.
problem Outlier contamination and heavy-tailed rewards in reinforcement learning.
method Develops a fully online robust policy evaluation procedure and efficient statistical inference.
result Establishes the Bahadur-type representation of the estimator and develops an online inference procedure.
Paper optimizes financial trading strategies under uncertain market conditions.
problem Guaranteeing robust positive expected profits in financial systems.
method Transformed semi-infinite constraints into structured policies and proposed a novel graphical approach.
result Demonstrated superior risk-adjusted returns and downside risk compared to conventional strategies.
New algorithm improves deep learning stability with limited data.
problem Stability and robustness in reinforcement learning with scarce data.
method Uncertainty-aware trust region approach to policy optimization.
result Stable policy updates adapt to uncertainty levels during learning.
New method estimates policy performance under unobserved confounding.
problem Estimating policy performance when decisions depend on unobserved variables.
method Developed worst-case bounds for robust OPE under unobserved confounding.
result Efficient procedure for computing worst-case bounds, proving statistical consistency.
Proposes fair and robust methods for estimating treatment effects.
problem Estimating treatment effects while maintaining fairness.
method Simple, nonparametric framework with fairness constraints.
result Estimators are double robust and characterize welfare trade-offs.
New trading policies preserve robust gains in presence of transaction costs.
problem Maintaining robust gains in asset trading with transaction costs.
method Proposed double linear trading policies, analyzed with Monte Carlo simulations and historical data.
result Desired robust positive expected gain can be preserved under certain conditions.
Robust Policy Search is the problem of learning policies that do not degrade in performance when subject to unseen environment model parameters. It is particularly relevant for transferring policies learned in a simulation environment to the real world. Several existing approaches involve sampling large batches of traj…