Presents SPEED, an algorithm for optimal policy evaluation in linear bandits with heteroscedastic noise.
problem Optimal data collection for policy evaluation in linear bandits with heteroscedastic reward noise.
method Formulated an optimal design for weighted least squares estimates, derived the optimal sample allocation, introduced SPEED algorithm, and derived regret bounds.
result SPEED leads to policy evaluation with MSE comparable to oracle strategy and significantly lower than random policy execution.
New method optimizes treatment policies to avoid winner's curse.
problem Winner's curse in treatment policy optimization.
method Inference-aware policy optimization.
result Optimizes for both estimated performance and downstream evaluation.
Paper addresses offline policy evaluation in RL, achieving near-optimal bounds for various policy classes.
problem Evaluate all policies in a class simultaneously for offline RL.
method Uniform convergence in OPE for various policy classes, achieving optimal episode complexity.
result Achieves optimal episode complexity of O(H^3/d_mε^2) for identifying ε-optimal policies.
New method efficiently evaluates policies using trajectory data.
problem Statistically efficient policy evaluation with limited data.
method Trajectory-based approach for policy evaluation.
result Improved sample complexity for policy evaluation.
Develops a method to estimate optimal policy value in online learning.
problem Challenges in evaluating ongoing policies in online learning environments.
method Doubly Robust Interval Estimation (DREAM) method.
result Valid inference on online conditional mean estimator with asymptotically normal distribution.
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
EPIC quantifies reward differences without policy optimization.
problem Distinguishing reward function quality from policy optimization issues.
method EPIC distance to compare reward functions directly.
result EPIC bounds policy training success and regret.
Proposes a conservative exploration method for RL agents.
problem Guaranteeing performance of exploratory policies in RL.
method Importance sampling for off-policy policy evaluation.
result Derives a regret bound ensuring no conservative constraint violation.
Paper tackles policy selection with logged data and limited online interactions.
problem Safe evaluation and deployment of offline reinforcement learning policies.
method Active offline policy selection combining logged data with online interaction.
result Improves upon state-of-the-art OPE estimates and pure online policy evaluation.
New methods for evaluating and optimizing policies in offline RL with unobserved confounders.
problem Evaluating and optimizing policies in the presence of unobserved confounders.
method Characterized settings and algorithms for consistent value estimates and lower bounds, with sample complexity guarantees.
result Proved local convergence guarantees for offline policy improvement.
Study minimax off-policy evaluation in multi-armed bandits with known and unknown behavior policies.
problem Evaluate policies in multi-armed bandits with unknown behavior policies.
method Develop minimax rate-optimal procedures for known and unknown behavior policies, including the Switch estimator and Chebyshev polynomial-based estimator.
result Plug-in estimator achieves optimal competitive ratio up to a logarithmic factor when behavior policy is unknown.
Optimizes insurance pricing by accounting for policyholders' price sensitivity.
problem Traditional insurance pricing does not consider policyholders' price sensitivity.
method Formulates insurance pricing as a decision-making problem and uses off-policy evaluation and stochastic control.
result Neural networks outperform existing techniques for policy optimization.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
Policy evaluation is a key process in Reinforcement Learning (RL). It assesses a given policy by estimating the corresponding value function. When using parameterized value functions, common approaches minimize the sum of squared Bellman temporal-difference errors and receive a point-estimate for the parameters. Kalman…
Study designs logging policies to minimize off-policy evaluation error.
problem Minimizing OPE error with logging policies for target policies.
method Characterizes reward-coverage tradeoff, proposes a unifying framework, derives optimal policies.
result Provides actionable guidance for firms choosing recommendation systems.
Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.
problem Quantifying uncertainty in off-policy estimates for safe deployment of target policies.
method Distributionally robust optimization for creating confidence bounds.
result Non-asymptotic and asymptotic guarantees for robust cumulative reward estimates.
This paper studies the statistical theory of batch data reinforcement learning with function approximation. Consider the off-policy evaluation problem, which is to estimate the cumulative value of a new target policy from logged history generated by unknown behavioral policies. We study a regression-based fitted Q iter…
We study the problem of policy evaluation and learning from batched contextual bandit data when treatments are continuous, going beyond previous work on discrete treatments. Previous work for discrete treatment/action spaces focuses on inverse probability weighting (IPW) and doubly robust (DR) methods that use a reject…
We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what …
Paper improves bootstrapping for off-policy reinforcement learning inference.
problem Improving bootstrapping for off-policy reinforcement learning inference.
method Proposes a bootstrapping FQE method for off-policy statistical inference and a subsampling procedure to improve runtime.
result Asymptotically efficient and distributionally consistent bootstrapping FQE method for off-policy inference.
Adaptive exploration scheme for evaluating multiple policies with different rewards.
problem Online multi-reward multi-policy evaluation.
method Adapted (ε,δ)-PAC perspective and MR-NaS exploration scheme to minimize sample complexity. result Demonstrated effectiveness of adaptive exploration in tabular domains.
Taylor expansions improve reinforcement learning policies.
problem Improving reinforcement learning policy optimization.
method Taylor expansion policy optimization.
result Taylor expansions enhance performance of distributed algorithms.
New methods improve temporal difference learning for policy evaluation in Markov decision processes.
problem Improving temporal difference learning for policy evaluation in Markov decision processes.
method Introduced variance-reduced forms of stochastic approximation to achieve non-asymptotic, instance-dependent optimality.
result Temporal difference learning is strictly suboptimal, but variance-reduced forms achieve optimality up to logarithmic factors.
Paper proposes a sequential statistical test for comparing imitation learning policies with near-optimal stopping.
problem Challenges in rigorously comparing imitation learning policies due to small sample sizes and potential p-hacking.
method Sequential statistical test that adapts the number of trials based on intermediate results, achieving near-optimal stopping.
result Reduces the number of evaluation trials by up to 32% compared to state-of-the-art baselines, saving significant time and effort.
A new estimator for evaluating policies in unknown environments.
problem Evaluating policies when both logging policy and value function are unknown.
method Doubly-Robust (DR) off-policy evaluation (OPE) estimator, DRUnknown, that estimates both the logging policy and value function.
result DRUnknown achieves the smallest asymptotic variance and is optimal when both models are correctly specified.
Paper tackles offline SSP with value iteration for policy evaluation and learning.
problem Goal-oriented RL with offline data and cost minimization.
method Simple value iteration algorithms for OPE and offline policy learning.
result Strong instance-dependent bounds implying near-minimax optimal worst-case bounds.
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…
New estimator improves policy evaluation in resource allocation RCTs.
problem Difficulty in evaluating policies optimizing limited resource allocation through RCTs.
method Proposes a novel estimator involving retrospective reshuffling of participants across experimental arms.
result The new estimator provides more accurate policy evaluations than common methods.
New approach for pricing evaluation improves on existing methods.
problem Improving off-policy evaluation for personalized pricing.
method Balanced policy evaluation framework with worst-case optimization.
result Empirical advantage over existing methods in pricing applications.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.
This study uses OPE methods to quickly assess auction policies.
problem Rapid decision-making in dynamic auction environments.
method Off-Policy Evaluation and counterfactual methods.
result Improved policy selection and optimization.
Entropy-regularized NPG methods converge linearly in discounted MDPs.
problem Theoretical limitations of NPG methods in reinforcement learning.
method Entropy regularization in conjunction with NPG methods for discounted MDPs.
result Entropy-regularized NPG methods converge linearly in discounted MDPs.
We consider off-policy evaluation and optimization with continuous action spaces. We focus on observational data where the data collection policy is unknown and needs to be estimated. We take a semi-parametric approach where the value function takes a known parametric form in the treatment, but we are agnostic on how i…
New methods improve off-policy evaluation for survival outcomes with censoring.
problem Systematic underestimation of policy performance due to censoring bias in survival outcomes.
method Proposes IPCW-IPS and IPCW-DR to handle censoring bias in survival outcomes.
result The proposed methods are unbiased and achieve double robustness.
This work develops confidence intervals for off-policy evaluation.
problem Estimating expected reward with uncertainty quantification.
method Primal-dual optimization with kernel Bellman loss and martingale concentration inequality.
result Developed practical algorithm for non-asymptotic confidence intervals.
We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electroni…
Monotonic policy improvement and off-policy learning are two main desirable properties for reinforcement learning algorithms. In this paper, by lower bounding the performance difference of two policies, we show that the monotonic policy improvement is guaranteed from on- and off-policy mixture samples. An optimization …
Off-policy evaluation of sequential decision policies from observational data is necessary in applications of batch reinforcement learning such as education and healthcare. In such settings, however, unobserved variables confound observed actions, rendering exact evaluation of new policies impossible, i.e., unidentifia…
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
New RL method learns value function for many policies using few key states.
problem Evaluate and improve policies in continuous control problems.
method Combines actor-critic architecture and policy embedding to learn a single value function for many policies.
result Value function minimizes prediction error by learning a small set of 'probing states' and their impact on policies' returns.
New algorithm optimally evaluates policies with linear approximations.
problem Policy evaluation with linear function approximation.
method Accelerated, variance-reduced fast temporal difference algorithm (VRFTD).
result VRFTD matches both deterministic and stochastic lower bounds.
The paper tackles robust reinforcement learning with performance guarantees.
problem Finding a robust policy for RMDP with state space uncertainties.
method Proposes RLSPI algorithm for learning optimal robust policy with performance bounds.
result Demonstrates the performance of RLSPI on standard benchmark problems.
Evaluating novel contextual bandit policies using logged data is crucial in applications where exploration is costly, such as medicine. But it usually relies on the assumption of no unobserved confounders, which is bound to fail in practice. We study the question of policy evaluation when we instead have proxies for th…
Develops CLTs for Markov chain transition probabilities and policies.
problem Estimating transition probabilities and policies in controlled Markov chains.
method Non-parametric estimator for transition matrices; CLTs for value, Q-, and advantage functions; goodness-of-fit tests.
result Asymptotic normality of estimators under specific logging policies.
Stochastic variance-reduced gradient (SVRG) is an optimization method originally designed for tackling machine learning problems with a finite sum structure. SVRG was later shown to work for policy evaluation, a problem in reinforcement learning in which one aims to estimate the value function of a given policy. SVRG m…
Reduces variance in noisy social outcomes to improve policy evaluation and optimization.
problem Improving access to opportunity through personalized treatment decisions.
method Data-driven dimensionality-reduction using reduced rank regression to denoise multiple outcomes.
result Improves estimation error in policy evaluation and optimization, including on real-world data.
We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality. We summarize how this duality may be applied to a variety of reinforcement learning (RL) settings, including policy evaluation or optimization, online or offline learning, and discounted or undisco…