Discuss new policy learning objectives and methods.
problem Improving policy learning efficiency and robustness.
method Introducing curvature considerations and calibration data methods.
result Efficient retargeting and distributionally robust policies.
Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
New method reduces over-pessimism in Bayesian control under parameter uncertainty.
problem Over-pessimism in Bayesian control due to misspecified priors.
method Distributionally robust Bayesian control (DRBC) with strong duality and optimization.
result Validated algorithm on synthetic and real data, reducing over-pessimism.
Paper tackles robust policy learning with incomplete data.
problem Learning policies from past data assumes future environment is the same, which is often false.
method Develops a distributionally robust policy evaluation and learning algorithm.
result Proposed algorithm provides robustness to adversarial perturbations and covariate shifts.
This paper tackles robust policy learning under concept drifts, improving upon existing methods.
problem Tackles robust policy learning under concept drifts, improving upon existing methods.
method Develops a doubly-robust estimator and a learning algorithm to maximize policy value within a given policy class.
result The proposed algorithm achieves sub-optimality gap of the order κ ( Π ) n − 1 / 2 κ(Π)n^{-1/2} κ ( Π ) n − 1/2 , demonstrating substantial improvement over existing benchmarks. This paper tackles RL issues with robust policies using historical data.
problem Limited data and mismatch between training and testing environments.
method Distributionally robust offline RL with linear function approximation.
result Achieved error bounds for sample complexity in RL.
A method for robust reinforcement learning in large state spaces.
problem Challenges in RL with large state spaces, costly data, and real-world dynamics deviation.
method Distributionally robust Markov decision processes with Gaussian Processes and maximum variance reduction.
result Efficient learning of multi-output nominal transition dynamics with statistical sample complexity bounds.
Proposes DR algorithms for distributionally robust off-policy evaluation and learning.
problem Sensitive to environment distribution shifts in offline observational data.
method Doubly robust and distributionally robust approaches for OPE/L.
result Achieves semiparametric efficiency and fast regret rate.
Real-world applications require RL algorithms to act safely. During learning process, it is likely that the agent executes sub-optimal actions that may lead to unsafe/poor states of the system. Exploration is particularly brittle in high-dimensional state/action space due to increased number of low-performing actions. …
Study robust control for systems with continuous states using adversarial perturbations.
problem Fragile policies in Markov control models under internal or external perturbations.
method Distributionally robust stochastic control with adaptive adversarial perturbations.
result Optimal robust policies for continuous state systems with uniform learning guarantees.
A new policy learning method allows policies to abstain when uncertain, improving safety and applicability.
problem Risk of making decisions without full confidence in uncertain predictions.
method Policy learning with abstention, identifying near-optimal policies and constructing an abstention rule.
result Improved safety and applicability in policy learning, with theoretical guarantees.
Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.
problem Quantifying uncertainty in off-policy estimates for safe deployment of target policies.
method Distributionally robust optimization for creating confidence bounds.
result Non-asymptotic and asymptotic guarantees for robust cumulative reward estimates.
New algorithm improves RL performance across different environments.
problem Improving reinforcement learning performance across various environments.
method Designing a fully model-free DRRL algorithm that learns from a single trajectory.
result Demonstrates superior robustness and sample efficiency compared to existing methods.
DR-RPO optimizes robust policies in RL with limited interaction, achieving sublinear regret.
problem Policy optimization in RL under distribution shift and adversarial dynamics.
method DR-RPO algorithm incorporating reference-policy regularization and upper confidence bonus for exploration.
result DR-RPO achieves sublinear regret and polynomial suboptimality bounds in robust RL.
A new algorithm improves offline reinforcement learning robustness.
problem Finding optimal policies in perturbed environments from offline data.
method Doubly Pessimistic Model-based Policy Optimization (P^2MPO) framework.
result Proves sample efficiency with robust partial coverage data.
New framework for robust reinforcement learning policies in uncertain environments.
problem Robust reinforcement learning policies in environments with distributional shifts.
method Comprehensive modeling framework centered around robust Markov decision processes (RMDPs).
result Existence and conditions for the dynamic programming principle (DPP) in RMDPs.
This work bridges offline RL and DRL to address distributional shift.
problem Distributional shift in offline RL due to difference in state-action visitation distributions.
method Proposes offline RL algorithms using DRL framework, characterizes sample complexity under single policy concentrability.
result Demonstrates superior performance of proposed algorithms through simulations.
Improved RL algorithm for robustness against parameter mismatches.
problem Learning robust control policies against parameter mismatches between training and testing environments.
method Formulated as DR-RL problem, proposed RPVL algorithm for tabular episodic learning with four divergences.
result Achieved i l d e O ( ∣ S ∣ ∣ A ∣ H 5 ) ilde{\mathcal{O}}(|\mathcal{S}||\mathcal{A}| H^{5}) i l d e O ( ∣ S ∣∣ A ∣ H 5 ) sample complexity uniformly better than existing results. A robust machine learning approach forecasts U.S. Treasury yields, reducing risk for investors.
problem Noisy and uncertain U.S. Treasury yields pose risk to forecast users.
method Formulates yield curve forecasting as a distributionally robust problem, combining factor models and machine learning.
result Robust forecast combinations improve out-of-sample performance across different maturity periods.
New algorithms learn robust policies from shifted distributions.
problem Learning robust policies in environments with distributional shifts.
method Two novel model-free algorithms: distributionally robust Q-learning and variance-reduced distributionally robust Q-learning.
result Achieves minimax sample complexity upper bound of i l d e O ( ∣ S ∣ ∣ A ∣ ( 1 − γ ) − 4 ε − 2 ) ilde O(|\mathbf{S}||\mathbf{A}|(1-γ)^{-4}ε^{-2}) i l d e O ( ∣ S ∣∣ A ∣ ( 1 − γ ) − 4 ε − 2 ) . A new framework for robust policy learning in MDPs with linear mixture dynamics.
problem Off-dynamics challenge in real-world decision-making problems.
method Linear mixture DRMDP framework, meta algorithm for robust policy learning.
result The new framework provides a more refined representation of uncertainties.
We develop a robust RL algorithm for off-dynamics environments with improved suboptimality bounds and computational efficiency.
problem Learning policies robust to uncertainties in transition dynamics between training and deployment environments.
method Distributionally robust Markov decision processes (DRMDPs) with a novel algorithm We-DRIVE-U.
result Improved suboptimality bound of O ~ ( d H ⋅ min { 1 / ρ , H } / K ) \widetilde{\mathcal{O}}\big({d H \cdot \min \{1/ρ, H\}/\sqrt{K} }\big) O ( d H ⋅ min { 1/ ρ , H } / K ) , near-optimal up to O ( H ) \mathcal{O}(\sqrt{H}) O ( H ) . Improves robust transfer learning with side information.
problem Addressing environmental shift in MDPs with side information.
method Estimate-centered uncertainty sets with side information integration.
result Improved robust policy with reduced sub-optimality gap.
Paper tackles robust optimization under uncertainty using nested distance.
problem Optimizing under distributionally robust uncertainty with nested distance.
method Equivalent recursive and dynamic programming reformulations for tractable optimization.
result Optimal robust policies can be found efficiently using convex optimization.
A new algorithm for deep Q-learning with robustness to state transition uncertainty.
problem Model uncertainty in state transitions for non-tabular, continuous state spaces.
method Distributionally robust approach using worst-case transition ball and dualized Bellman operator with Sinkhorn distance.
result Optimal policy found through solving non-linear Bellman equation with neural network parameterization.
Paper studies S-rectangular DR-RL models for robust reinforcement learning with near-optimal sample complexity.
problem Addressing distributional discrepancies in reinforcement learning environments.
method Empirical value iteration algorithm for divergence-based S-rectangular DR-RL models.
result Near-optimal sample complexity bound of O ( ∣ S ∣ ∣ A ∣ ( 1 − γ ) − 4 ε − 2 ) O(|\mathcal{S}||\mathcal{A}|(1-γ)^{-4}\varepsilon^{-2}) O ( ∣ S ∣∣ A ∣ ( 1 − γ ) − 4 ε − 2 ) . Develops robust learning framework under distributional perturbations.
problem Learning robust to data distributional changes.
method Distributionally Robust Optimization (DRO) under Wasserstein metric.
result Establishes performance guarantees and tractable formulations.
New algorithms tackle robust RL with linear models, revealing unique challenges.
problem Distributionally robust offline RL with uncertainty in dynamics.
method Proposes minimax optimal and computationally efficient algorithms using novel function approximation mechanisms.
result Function approximation in robust offline RL is distinct and harder than in standard offline RL.
Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
This work tackles robust RL in multi-agent settings, improving sample efficiency.
problem Overcoming environmental uncertainties in multi-agent reinforcement learning.
method Proposes DRNVI, a sample-efficient algorithm for learning robust equilibria in RMGs.
result Establishes near-optimal sample complexity for solving RMGs.
Paper tackles robust reinforcement learning with minimal data.
problem Learning robust policies from limited data in uncertain environments.
method Distributionally robust formulation, model-based algorithm combining value iteration and pessimism.
result Proves near-optimal sample complexity for robust offline RL.
Proposes robust ITRs integrating multiple datasets to handle posterior shift.
problem Posterior shift in conditional outcome distributions between source and target populations.
method Distributionally robust approach with closed-form solution and adaptive uncertainty tuning.
result Achieves superior performance compared to existing methods in simulations and real-data applications.
New RL method tackles sim-to-real gap using interactive data collection.
problem Sim-to-real gap in reinforcement learning.
method Distributionally robust reinforcement learning with interactive data collection.
result Proves sample-efficient learning is impossible without additional assumptions.
STEEL tackles batch RL with singularity, improving policy optimization.
problem Existing RL methods assume absolutely continuous data, but STEEL handles non-overlapping regions.
method Proposes STEEL algorithm using maximum mean discrepancy and distributionally robust optimization.
result First finite-sample regret guarantee for batch RL with singularity.
Study improves reinforcement learning for stable long-term performance.
problem Distributionally robust average-reward reinforcement learning for stable long-term performance.
method Proposes two algorithms to achieve near-optimal sample complexity.
result Achieves a sample complexity of O ( ∣ S ∣ ∣ A ∣ t m i x 2 ε − 2 ) O(|\mathbf{S}||\mathbf{A}| t_{\mathrm{mix}}^2\varepsilon^{-2}) O ( ∣ S ∣∣ A ∣ t mix 2 ε − 2 ) for estimating optimal policy and robust average reward. This work improves autonomous racing by creating diverse opponents and adapting risk.
problem Balancing performance and safety in autonomous racing environments.
method Developed a self-play method using replica-exchange Markov chain Monte Carlo for diverse opponents and a distributionally robust bandit optimization for adaptive risk adjustment.
result Demonstrated real-time motion-planning methods achieving speeds comparable to Formula One racecars.
Paper tackles offline preference-based RL with human feedback.
problem Offline Preference-based Reinforcement Learning with preference feedback.
method Two-step approach: MLE for reward estimation and distributionally robust planning.
result First guarantee for learning any target policy with polynomial samples.
New scalable methods for robust model learning from large datasets.
problem Training robust models resistant to data distribution shifts.
method Composite optimization for distributionally robust optimization (DRO).
result Scalable methods for learning robust models from large datasets.
New approach improves model generalization through distributionally robust learning.
problem Improving model generalization in machine learning.
method Stochastic gradient descent applied to the outer minimization problem, with gradient estimation through multi-level Monte Carlo randomization.
result Our approach yields significant benefits over previous work in numerical experiments.
DRO-REBEL improves LLM alignment by robustly updating models online.
problem Overfitting and drifting of LLMs during RLHF.
method DRO-REBEL uses type- p p p Wasserstein, KL, and χ 2 χ^2 χ 2 ambiguity sets for robust online updates. result DRO-REBEL achieves faster convergence and better performance than prior methods.
Approximate dynamic programming is a popular method for solving large Markov decision processes. This paper describes a new class of approximate dynamic programming (ADP) methods- distributionally robust ADP-that address the curse of dimensionality by minimizing a pessimistic bound on the policy loss. This approach tur…
New regularizer for machine learning using private data.
problem Machine learning with private data.
method Distributionally-robust optimization with locally-differentially-private datasets.
result New regularizer for training linear regression models.
The paper connects three machine learning methods to reduce generalization errors.
problem Reducing generalization errors in machine learning models.
method Distributionally robust optimization, Bayesian methods, and regularization.
result Machine learning models can be characterized using distributional uncertainty and robustness measures.
A general duality proof for Wasserstein distributionally robust optimization.
problem Optimizing under uncertainty with Wasserstein distance.
method One-dimensional convex analysis and interchangeability principle.
result General duality result holds for various distributions and costs.
Improved estimator reduces bias in statistical learning models.
problem Asymptotic bias in classic WDRO estimator.
method Adjusted Wasserstein distributionally robust estimator.
result Asymptotic unbiased estimator with smaller MSE.
Paper develops a robust Bayesian optimization method for noisy zeroth-order settings.
problem Achieving robustness to distributional shift in machine learning.
method Distributionally robust Bayesian optimization (DRBO) algorithm for noisy zeroth-order optimization.
result DRBO algorithm provably obtains sub-linear robust regret in various settings.
Wasserstein distributionally robust optimization estimators are obtained as solutions of min-max problems in which the statistician selects a parameter minimizing the worst-case loss among all probability models within a certain distance (in a Wasserstein sense) from the underlying empirical measure. While motivated by…
Algorithm learns robust equilibrium in online Markov games with interactive data.
problem Sim-to-real gap in reinforcement learning.
method Distributionally robust RL with minimum value assumption, least square value iteration.
result Sample-efficient algorithm for robust equilibrium in online Markov games.