New framework studies policy learning problems under data scarcity.
problem Learning improving policies when data is insufficient.
method Developed a mathematical framework for policy learning problems.
result Reduced policy learning problems to simpler ones in sample complexity.
Standard reinforcement learning methods aim to master one way of solving a task whereas there may exist multiple near-optimal policies. Being able to identify this collection of near-optimal policies can allow a domain expert to efficiently explore the space of reasonable solutions. Unfortunately, existing approaches t…
We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our…
Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradien…
A new estimator for evaluating policies in unknown environments.
problem Evaluating policies when both logging policy and value function are unknown.
method Doubly-Robust (DR) off-policy evaluation (OPE) estimator, DRUnknown, that estimates both the logging policy and value function.
result DRUnknown achieves the smallest asymptotic variance and is optimal when both models are correctly specified.
This paper bridges the gap between theoretical and practical OPE for bandit problems.
problem Estimating the value of a target policy from samples generated by different policies.
method Categorizing OPE situations based on evaluation policy properties, proposing a meta-algorithm.
result Meta-algorithm successfully bridges the gap between theoretical and practical OPE for bandit problems.
Step-DAD improves BED by periodically updating a design policy during experiments.
problem Improving flexibility and robustness in Bayesian experimental design.
method Semi-amortized, policy-based approach that updates a design policy during data collection.
result Consistently superior decision-making and robustness compared to current BED methods.
Pessimistic estimator improves multi-objective policy optimization.
problem Optimizing multi-objective policies from existing data.
method Pessimistic estimator based on inverse propensity scores (IPS).
result Pessimistic estimator outperforms naive IPS estimator in theory and experiments.
Simplifies RL training with fewer techniques, reducing bias and instability.
problem Training instabilities and high sample complexity in RL.
method Introduced a simple deterministic policy gradient, used propensity estimation, and delayed policy updates.
result Improved performance and reduced sample complexity through these techniques.
This paper tackles robust policy learning under concept drifts, improving upon existing methods.
problem Tackles robust policy learning under concept drifts, improving upon existing methods.
method Develops a doubly-robust estimator and a learning algorithm to maximize policy value within a given policy class.
result The proposed algorithm achieves sub-optimality gap of the order κ(Π)n−1/2, demonstrating substantial improvement over existing benchmarks. Adaptive reduction scheme approximates optimal policy in regularized MDPs.
problem Finding near optimal policy in regularized MDPs with biased solutions.
method Adaptive reduction of regularization parameter λ to approximate optimal policy.
result Iteration complexity reduced for obtaining ε-optimal policy.
Transfer Learning (TL) has shown great potential to accelerate Reinforcement Learning (RL) by leveraging prior knowledge from past learned policies of relevant tasks. Existing transfer approaches either explicitly computes the similarity between tasks or select appropriate source policies to provide guided explorations…
We study the problem of off-policy evaluation (OPE) in Reinforcement Learning (RL), where the aim is to estimate the performance of a new policy given historical data that may have been generated by a different policy, or policies. In particular, we introduce a novel doubly-robust estimator for the OPE problem in RL, b…
Develops methods to estimate and quantify uncertainty in off-policy evaluation.
problem Uncertainty quantification in off-policy evaluation for new policy deployment.
method Designs a pseudo policy to generate subsamples and applies conformal prediction.
result Valid interval estimators for target policy's return with uncertainty quantification.
End-to-end policy learning method improves CATE estimation.
problem Learning optimal treatment policies from partially observed data.
method Modified causal forest for policy learning.
result Maximizing policy value is equivalent to minimizing CATE.
Improves pre-trial risk assessments by making them safer without changing existing rules.
problem Improving pre-trial risk assessments while maintaining deterministic rules.
method Developed a maximin robust optimization approach to find a safer policy.
result Can safely improve certain components of the risk assessment instrument.
New bounds assess policy evaluation under unobserved confounders, showing model-based methods are more effective.
problem Policy evaluation under unobserved confounders in uncertain causal environments.
method Developed worst-case bounds for sensitivity to unobserved confounders, demonstrating model-based methods are more effective.
result Model-based approaches with robust MDPs provide sharper lower bounds for policy evaluation.
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.
Improves policies with high certainty, even in small samples.
problem Ensuring new policies are better than the baseline with high probability.
method Leverages powerful safety tests and multiple testing for threshold policies.
result Controls the rate of adopting a worse policy to pre-specified error level.
Develops DPG methods for continuous-time RL with deterministic policies.
problem High variance and slow convergence in stochastic policy RL methods.
method Derives continuous-time policy gradient formula and proposes CT-DDPG algorithm.
result CT-DDPG achieves superior stability and faster convergence in continuous-time RL.
We study a reinforcement learning setting, where the state transition function is a convex combination of a stochastic continuous function and a deterministic function. Such a setting generalizes the widely-studied stochastic state transition setting, namely the setting of deterministic policy gradient (DPG). We firstl…
We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-policy evaluation (OPE). Starting from the doubly robust (DR) estimator (Jiang & Li, 2016), we provid…
New method for estimating value of optimal policies in uncertain scenarios.
problem Inference for optimal policies when they are non-unique or nearly deterministic.
method Semiparametric efficiency bound, uniformly weighted estimator, NSAVE method.
result Proposes NSAVE method for robust inference in uncertain optimal policies.
A new method for MARL with partial observations reduces communication overhead.
problem Inefficient MARL algorithms in large-scale problems due to state and action information sharing.
method Distributed zeroth-order policy optimization with local policy gradient estimation using consensus.
result The method converges to a policy that is a stationary point of the global objective function.
Extends OPE to evaluate policies using diverse logging data.
problem Evaluate policies using log data from different policies.
method Develops an OPE method for various logging policies.
result Method's predictions converge to true performance as sample size increases.
Paper enhances RL policies using trust region optimization for offline data.
problem Limited data in offline RL settings.
method Trust region optimization for value enhancement.
result Enhanced policy values with faster convergence.
Analyzes securitization impacts on monetary and fiscal policies.
problem Impact of securitization on monetary and fiscal policies.
method Develops optimal conditions, identifies constraints, introduces new decision models.
result Identifies constraints and interactions of securitization with capital-reserve requirements.
ReSkill reconciles RL skill creation with policy optimization.
problem RL policies lack reusable strategies across tasks.
method Integrates skill creation into RL loop with three mechanisms.
result Consistently outperforms existing methods, especially on unseen tasks.
MoMA improves model-based RL by using unrestricted policy classes.
problem Limited sample efficiency and generalizability in model-based offline RL.
method Model-based mirror ascent algorithm with general function approximations.
result Theoretical guarantees and practical implementation of MoMA.
Advocates focusing on utility functions to avoid unfair outcomes.
problem Unfair outcomes from fairness criteria in optimizing policies.
method Defines value of information fairness and proposes modifying utility functions.
result Value of information fairness leads to better answers than existing fairness notions.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.
A new method reduces variance in PG methods for RL, improving efficiency and convergence.
problem Improving sample efficiency and convergence of policy gradient methods in reinforcement learning.
method Proposes a gradient truncation mechanism and designs TSIVR-PG method to maximize rewards and utility.
result Shows sample complexity of TSIVR-PG to find ε-stationary policy and global ε-optimal policy.
Benchmarking off-policy evaluation methods for complex policies.
problem Lack of comprehensive benchmarks for off-policy evaluation methods.
method Collection of challenging high-dimensional control tasks and datasets.
result Standardized measure of progress for OPE methods.
We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are com…
CPR models complex decision processes by breaking them into context-specific policies, improving interpretability and accuracy.
problem Interpreting dynamic human decision-making processes in medical contexts.
method Develops Contextualized Policy Recovery (CPR) framework for multi-task learning, modeling each context-specific policy as a linear map.
result Achieves state-of-the-art performance in predicting medical decisions, closing the gap between interpretable and black-box methods.
Improved off-policy selection and learning in contextual bandits with better guarantees.
problem Selecting or training a reward-maximizing policy using data from a fixed behavior policy.
method A betting-based confidence bound applied to an inverse propensity weight sequence for off-policy selection, and a freezing condition for off-policy learning.
result The proposed methods achieve significantly improved guarantees over prior work, especially in small-data regimes.
We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electroni…
When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a colle…
Social Security and other public policies can be viewed as a series of cash in and outflows that depend on parameters such as the age distribution of the population and the retirement age. Given forecasts of these parameters, policies can be designed to be financially stable, i.e., to terminate with a zero balance. If …
A key problem in reinforcement learning for control with general function approximators (such as deep neural networks and other nonlinear functions) is that, for many algorithms employed in practice, updates to the policy or Q-function may fail to improve performance---or worse, actually cause the policy performance …
New estimator GMIPS reduces variance in ranking policy evaluation.
problem High variance in off-policy evaluation for ranking policies.
method GMIPS estimator with user behavior model on ranking embedding spaces.
result GMIPS achieves lowest MSE and balances bias-variance trade-off.
Robust Policy Search is the problem of learning policies that do not degrade in performance when subject to unseen environment model parameters. It is particularly relevant for transferring policies learned in a simulation environment to the real world. Several existing approaches involve sampling large batches of traj…
Value aggregation is a general framework for solving imitation learning problems. Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting. While the existence of a good policy in the policy sequence can be guarant…
We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …
Sayer uses implicit feedback to optimize system policies.
problem Leveraging implicit feedback to improve system policies is difficult due to bias and incompleteness.
method Sayer combines randomized exploration and unbiased counterfactual estimators to evaluate and train new policies using implicit feedback.
result Sayer can accurately evaluate and train new policies that outperform existing ones.
Designs a single policy for collecting data to train near-optimal policies.
problem Engineering overhead in deploying minimax procedures for stochastic linear contextual bandits.
method Designs a single stochastic policy to collect data from which a near-optimal policy can be extracted.
result The designed policy can collect data from which a near-optimal policy can be extracted.
Improving the sample efficiency in reinforcement learning has been a long-standing research problem. In this work, we aim to reduce the sample complexity of existing policy gradient methods. We propose a novel policy gradient algorithm called SRVR-PG, which only requires O(1/ε3/2) episodes to find an ε-approxima…