New method estimates state-action stationary distribution for better off-policy policy evaluation.
problem Accurately estimating state-action stationary distribution for off-policy policy evaluation.
method Estimated Mixture Policy (EMP) for state and state-action stationary distribution corrections.
result Empirical validation shows improved accuracy over state-of-the-art methods.
New method extends fitted Q-evaluation for distributional off-policy reinforcement learning.
problem Estimating return distribution in reinforcement learning using offline data.
method Developed a set of guiding principles and new FDE methods with theoretical justification.
result FDE methods outperform existing approaches in simulations and real-world games.
Paper addresses off-policy evaluation and learning with covariate shift.
problem Evaluating and training a new policy using historical data with a covariate shift.
method Derives efficiency bounds and proposes doubly robust estimators for OPE and OPL under covariate shift.
result Proposes estimators for off-policy evaluation and learning under covariate shift.
We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what …
Improved off-policy evaluation for MDPs with weak distributional overlap.
problem Evaluation of policies when target and data-collection distributions are not strongly overlapping.
method Truncated Doubly Robust (TDR) estimators for off-policy evaluation in MDPs under weak distributional overlap.
result TDR estimators can recover large-sample behavior and are consistent even when distribution ratios are not square-integrable.
New method for distributional off-policy evaluation using Bellman residual minimization.
problem Learning return distribution from offline data generated by a different policy.
method Energy Bellman Residual Minimizer (EBRM) method.
result Established finite-sample error bound for EBRM estimator.
Paper introduces DQPOPE for estimating return distributions in reinforcement learning.
problem Estimating the entire return distribution from off-policy data.
method Deep quantile process regression for distributional off-policy evaluation.
result DQPOPE achieves statistical advantages by estimating full return distribution with same sample size.
Unified framework for reliable uncertainty quantification in RL.
problem Uncertainty quantification in high-stakes reinforcement learning.
method Unified conformal prediction framework integrating distributional RL and conformal calibration.
result Significantly improved coverage and reliability over standard methods.
This paper optimizes off-policy evaluation in reinforcement learning with function approximation.
problem Estimating cumulative value of a new policy from logged data generated by an unknown policy.
method Regression-based fitted Q iteration method, equivalent to estimating conditional mean embedding of transition operator.
result The method is minimax-optimal, with nearly minimal estimation error.
Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights…
This paper proposes a method to evaluate policies using quantile metrics, improving upon existing mean-based approaches.
problem Evaluating policies using mean-based metrics ignores the variability of outcomes, especially in skewed reward distributions.
method The paper introduces a doubly-robust inference procedure for quantile off-policy evaluation using deep conditional generative learning.
result The proposed estimator outperforms classical OPE estimators for mean outcomes in heavy-tailed reward distributions.
Unified DICE estimators as regularized Lagrangians for improved off-policy evaluation.
problem Improving off-policy evaluation from behavior-agnostic data.
method Unified derivation of DICE estimators as regularized Lagrangians of a linear program.
result Dual solutions offer greater flexibility and provide superior estimates in practice.
The paper improves QD policy ensembles using distribution ratio estimators.
problem Training diverse and high-quality reinforcement learning agents.
method Using Stein variational gradient descent and distribution ratio estimators.
result The method generates diverse and high-quality reinforcement learning agents.
A new estimator improves off-policy evaluation in RL, outperforming existing methods.
problem Estimating performance of a new policy using historical data from a different policy.
method Doubly-robust estimator based on Targeted Maximum Likelihood Estimation, with variance reduction techniques.
result Our estimator uniformly outperforms existing methods across various RL environments and levels of model misspecification.
Paper improves bootstrapping for off-policy reinforcement learning inference.
problem Improving bootstrapping for off-policy reinforcement learning inference.
method Proposes a bootstrapping FQE method for off-policy statistical inference and a subsampling procedure to improve runtime.
result Asymptotically efficient and distributionally consistent bootstrapping FQE method for off-policy inference.
CPME embeds counterfactual outcomes in RKHS for flexible policy evaluation.
problem Estimating counterfactual policy outcomes for decision-making.
method Counterfactual Policy Mean Embedding (CPME) framework in RKHS, plug-in and doubly robust estimators, kernel test statistic.
result Doubly robust estimator improves convergence rates and asymptotic normality.
Paper develops robust policy evaluation for reinforcement learning with outlier and heavy-tailed rewards.
problem Outlier contamination and heavy-tailed rewards in reinforcement learning.
method Develops a fully online robust policy evaluation procedure and efficient statistical inference.
result Establishes the Bahadur-type representation of the estimator and develops an online inference procedure.
Paper addresses OPE for dependent bandit samples using MDS and batch updates.
problem Evaluating policies from non-i.i.d. historical data in contextual bandits.
method Constructs an MDS-based estimator for dependent samples, solves batch update and deficient support issues.
result Derives an asymptotically normal estimator for evaluation policy value.
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.
EPIC quantifies reward differences without policy optimization.
problem Distinguishing reward function quality from policy optimization issues.
method EPIC distance to compare reward functions directly.
result EPIC bounds policy training success and regret.
Validates policies using past observational data with guarantees about out-of-sample performance.
problem Evaluating decision policies using past data observed under a different policy.
method Sample-splitting method to draw inferences about the entire loss distribution with finite-sample coverage guarantees.
result Valid inferences about out-of-sample loss with finite-sample coverage guarantees, accounting for model misspecifications.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
Taylor expansions improve reinforcement learning policies.
problem Improving reinforcement learning policy optimization.
method Taylor expansion policy optimization.
result Taylor expansions enhance performance of distributed algorithms.
PEOC uses policy entropy to detect untrained states in RL.
problem Detecting untrained states in reinforcement learning for safety.
method Policy entropy based one-class classifier.
result PEOC is highly competitive and reliable.
New offline RL study shows exponential sample requirement for accurate policy evaluation.
problem Understanding statistical limits of offline RL with linear function approximation.
method Analyzes necessary representational and distributional conditions for sample-efficient offline reinforcement learning.
result Even with realizability and good feature coverage, offline RL requires exponential samples for accurate policy evaluation.
Unified framework for error quantification in off-policy evaluation with distributional shift.
problem Establishing high-confidence CI for target policy value from offline data.
method Unified error analysis quantifying misspecification and sampling errors.
result Achieves tightest possible CI and robustness against distributional shifts.
This paper improves reinforcement learning by estimating return distributions using quantiles.
problem Improving reinforcement learning by estimating return distributions.
method Quantile-based distributional reinforcement learning, using quantile-projected distributional Bellman equations.
result The quantile-based approach achieves optimal sample efficiency and asymptotic efficiency.
This paper improves reinforcement learning by estimating return distributions using quantiles.
problem Improving reinforcement learning by estimating return distributions.
method The paper uses quantile-based distributional reinforcement learning to characterize return distributions.
result The quantile-based approach achieves optimal sample efficiency and asymptotic efficiency.
Study designs logging policies to minimize off-policy evaluation error.
problem Minimizing OPE error with logging policies for target policies.
method Characterizes reward-coverage tradeoff, proposes a unifying framework, derives optimal policies.
result Provides actionable guidance for firms choosing recommendation systems.
This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-…
Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.
problem Quantifying uncertainty in off-policy estimates for safe deployment of target policies.
method Distributionally robust optimization for creating confidence bounds.
result Non-asymptotic and asymptotic guarantees for robust cumulative reward estimates.
We identify a fundamental problem in policy gradient-based methods in continuous control. As policy gradient methods require the agent's underlying probability distribution, they limit policy representation to parametric distribution classes. We show that optimizing over such sets results in local movement in the actio…
Study tackles OPE in confounded settings, estimating policy value from proxies.
problem Difficulty in OPE due to unobserved confounders in infinite-horizon RL.
method Two-stage approach: estimating stationary distribution ratios and combining optimal balancing.
result Policy value can be identified from off-policy data with proxies and latent variable model.
This paper tackles robust policy learning under concept drifts, improving upon existing methods.
problem Tackles robust policy learning under concept drifts, improving upon existing methods.
method Develops a doubly-robust estimator and a learning algorithm to maximize policy value within a given policy class.
result The proposed algorithm achieves sub-optimality gap of the order κ(Π)n−1/2, demonstrating substantial improvement over existing benchmarks. Paper introduces a new distributional successor measure for reinforcement learning.
problem Learning the distributional consequences of behavior in reinforcement learning.
method Formulates distributional successor measure as a distribution over distributions, proposes algorithm to learn it from data.
result Demonstrates zero-shot risk-sensitive policy evaluation.
Novel LSE estimator improves off-policy learning and evaluation.
problem High variance and poor performance with low-quality propensity scores and heavy-tailed reward distributions.
method Introduces a novel estimator based on the log-sum-exponential (LSE) operator.
result Achieves convergence rate of O(n−ε/(1+ε)) for regret bounds. SDM Policy accelerates inference for robotic tasks while maintaining high action quality.
problem Prolonged inference times in diffusion-based policies for high-frequency control tasks.
method Two-stage optimization: score matching and distribution matching; dual-teacher mechanism.
result 6x inference speedup with state-of-the-art action quality.
Sparse feature selection improves batch RL efficiency.
problem High-dimensional batch RL with many features.
method Sparse linear function approximation, Lasso, group Lasso, fitted Q-evaluation, fitted Q-iteration.
result Sparse feature selection makes batch RL more sample efficient.
New method for evaluating policies in complex decision-making models with hidden variables.
problem Evaluating policies in partially observable Markov decision processes with hidden confounders.
method Introduces novel identification methods and minimax estimation techniques for linking target policy's value and observed data distribution.
result Proposes three estimators for off-policy evaluation in POMDPs with latent confounders, demonstrating their effectiveness through nonasymptotic and asymptotic analysis.
Bayesian approach for modeling counterfactual distribution and off-policy evaluation.
problem Modeling the counterfactual distribution and off-policy evaluation.
method Bayesian conditional mean embeddings and novel Bayesian methods for estimating ultimate treatment effect.
result Quantifying epistemic uncertainty in the counterfactual distribution and off-policy evaluation.
We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves…
A novel fully asynchronous scheme for distributed reinforcement learning over networks.
problem Policy evaluation in distributed reinforcement learning over networks.
method Design of a stochastic average gradient (SAG) based distributed algorithm and push-pull augmented graph approach.
result The proposed algorithm converges at a linear rate of \(\mathcal{O}(c^k)\) with \(c\in(0,1)\) and \(k\) increasing by one per node update.
BCRL learns a Bellman complete representation for offline RL policy evaluation.
problem Learning a Q-function efficiently from offline data.
method BCRL learns a linear Bellman complete representation directly from data, enabling efficient OPE.
result BCRL achieves competitive OPE error and outperforms FQE in certain scenarios.
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
problem Estimating value function of a target policy from offline data collected by a behavior policy.
method Proposes VA-OPE, an algorithm that reweights Bellman residual using estimated variance of the value function.
result Achieves a tighter error bound than the best-known result.
A new method estimates multi-dimensional value distributions using Hilbert space embeddings.
problem Estimating value distributions in complex, multi-dimensional reinforcement learning settings.
method Hilbert space mappings and kernel mean embeddings to estimate the kernel mean embedding of multi-dimensional value distributions.
result Uniform convergence guarantees and robust off-policy evaluation demonstrated in simulations.
A new method for estimating causal effects using synthetic controls.
problem Evaluating causal effects of policy changes in settings with observational data.
method Distributional Synthetic Controls method.
result Allows construction of entire synthetic distributions for the treated unit.
In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correct…
Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.