We study the problem of off-policy critic evaluation in several variants of value-based off-policy actor-critic algorithms. Off-policy actor-critic algorithms require an off-policy critic evaluation step, to estimate the value of the new policy after every policy gradient update. Despite enormous success of off-policy …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Off-policy estimation”
New estimator uses clustering to improve off-policy evaluation accuracy.
problem Improving off-policy evaluation accuracy when logging and evaluation policies differ.
method Proposes an estimator that shares information across similar contexts using clustering.
result Clustering contexts improves estimation accuracy, especially in deficient information settings.
We study the problem of off-policy evaluation (OPE) in Reinforcement Learning (RL), where the aim is to estimate the performance of a new policy given historical data that may have been generated by a different policy, or policies. In particular, we introduce a novel doubly-robust estimator for the OPE problem in RL, b…
Novel LSE estimator improves off-policy learning and evaluation.
problem High variance and poor performance with low-quality propensity scores and heavy-tailed reward distributions.
method Introduces a novel estimator based on the log-sum-exponential (LSE) operator.
result Achieves convergence rate of for regret bounds.
The paper introduces a novel method for stable off-policy learning using value function chaining.
problem Stability issues in off-policy reinforcement learning.
method The approach involves learning on-policy first, then chaining off-policy value estimates.
result The method guarantees convergence and can approximate off-policy TD solutions.
Paper improves bootstrapping for off-policy reinforcement learning inference.
problem Improving bootstrapping for off-policy reinforcement learning inference.
method Proposes a bootstrapping FQE method for off-policy statistical inference and a subsampling procedure to improve runtime.
result Asymptotically efficient and distributionally consistent bootstrapping FQE method for off-policy inference.
Unified DICE estimators as regularized Lagrangians for improved off-policy evaluation.
problem Improving off-policy evaluation from behavior-agnostic data.
method Unified derivation of DICE estimators as regularized Lagrangians of a linear program.
result Dual solutions offer greater flexibility and provide superior estimates in practice.
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.
Paper proposes a framework for reliable off-policy evaluation in reinforcement learning.
problem Quantifying uncertainty in off-policy estimates for safe deployment of target policies.
method Distributionally robust optimization for creating confidence bounds.
result Non-asymptotic and asymptotic guarantees for robust cumulative reward estimates.
OSIRIS reduces variance in off-policy evaluation by omitting irrelevant states.
problem High variance in importance sampling-based OPE estimators.
method OSIRIS reduces variance by omitting likelihood ratios associated with states irrelevant to return.
result OSIRIS is unbiased and has lower variance than ordinary importance sampling.
We consider the problem of off-policy evaluation in Markov decision processes. Off-policy evaluation is the task of evaluating the expected return of one policy with data generated by a different, behavior policy. Importance sampling is a technique for off-policy evaluation that re-weights off-policy returns to account…
Memory-efficient algorithm reduces variance in off-policy RL.
problem High variance in off-policy policy optimization.
method Memory-efficient, stochastically variance-reduced algorithm using off-policy samples.
result Empirically validated effectiveness of the proposed algorithm.
Paper proposes a method to create more reliable confidence intervals for off-policy evaluations.
problem Creating reliable confidence intervals for off-policy evaluations.
method Proposes a deeply-debiasing procedure to construct efficient, robust, and flexible confidence intervals.
result Validated by theoretical results and numerical experiments, the method improves the reliability of off-policy evaluations.
Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights…
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Mark…
STAR framework reduces OPE variance by distilling complex problems into discrete ARPs.
problem High variance and bias in off-policy evaluation methods.
method STAR framework that includes various OPE estimators and leverages state abstraction.
result Predictions from ARPs estimated from off-policy data are asymptotically correct.
Paper proposes FPG algorithm for unbiased off-policy PG estimation.
problem Challenges in off-policy policy gradient estimation.
method Double Fitted PG estimation (FPG) algorithm for arbitrary policy parameterization.
result Empirically, FPG significantly outperforms existing methods.
New method estimates off-policy data without needing known behavior policy.
problem Long-horizon reinforcement learning with limited on-policy data.
method Formulates problem as fixed point of operator, uses RKHS for estimation.
result Asymptotic consistency and finite-sample generalization proven.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
Off-policy deep reinforcement learning (RL) algorithms are incapable of learning solely from batch offline data without online interactions with the environment, due to the phenomenon known as \textit{extrapolation error}. This is often due to past data available in the replay buffer that may be quite different from th…
DOLCE improves off-policy evaluation and learning by decomposing effects.
problem Bias in off-policy evaluation and learning due to policy mismatch.
method Uses lagged contexts and a moment-based training procedure to decompose and cancel bias.
result DOLCE achieves substantial improvements in off-policy evaluation and learning.
Concept-driven OPE reduces variance in off-policy decision evaluation.
problem High variance in off-policy decision evaluation due to limited sample sizes.
method Integrating human-explainable concepts into OPE to reduce variance.
result Concept-based OPE estimators remain unbiased and reduce variance when concepts are known and predefined.
New findings show optimization is crucial for OPL in large action spaces.
problem Challenges in optimizing policies for large action spaces in offline contextual bandits.
method Weighed log-likelihood objectives and estimator-aware policy parametrization.
result Simple weighted log-likelihood objectives enjoy better optimization properties and recover competitive policies.
Develops methods to estimate and quantify uncertainty in off-policy evaluation.
problem Uncertainty quantification in off-policy evaluation for new policy deployment.
method Designs a pseudo policy to generate subsamples and applies conformal prediction.
result Valid interval estimators for target policy's return with uncertainty quantification.
This paper unifies two types of statistical methods for estimating treatment effects.
problem Isolating online A/B-tests and off-policy evaluation.
method Establishes formal equivalence between online Difference-in-Means and off-policy Inverse Propensity Scoring methods.
result Standard online methods are mathematically equivalent to off-policy methods with optimal control variates.
We propose an estimator and confidence interval for computing the value of a policy from off-policy data in the contextual bandit setting. To this end we apply empirical likelihood techniques to formulate our estimator and confidence interval as simple convex optimization problems. Using the lower bound of our confiden…
Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributionscs.LG
This work provides guarantees for off-policy function estimation under realizability assumptions.
problem Estimating the value function of a policy under user-specified error-measuring distributions.
method The approach involves imposing a flexible regularization on the MIS objectives to account for an arbitrary user-specified distribution.
result Exact characterization of the optimal dual solution that determines the data-coverage assumption in the case of value-function learning.
Paper introduces risk assessment for contextual bandits without experiments.
problem Evaluate policies using logged data in context bandits.
method Lipschitz risk functionals and Off-Policy Risk Assessment (OPRA) framework.
result OPRA provides finite sample guarantees for various risk estimates.
Paper finds efficient OPE estimator for multiple logging policies with minimum variance.
problem Finding optimal importance sampling weights for multiple logging policies with varying variances.
method Established efficiency bound under stratified sampling and proposed an estimator achieving this bound.
result Proposed estimator achieves minimum variance for any instance.
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy …
FQE with deep neural networks achieves asymptotic normality and finite-sample bounds.
problem Theoretical understanding of FQE with general differentiable function approximators.
method Z-estimation theory applied to FQE with deep neural networks.
result FQE estimation error is asymptotically normal with explicit variance.
Proposes a method for obtaining interval bounds in off-policy evaluation.
problem Provides provably correct upper and lower bounds for off-policy evaluation.
method Searches for the maximum and minimum values of the expected reward among Lipschitz Q-functions.
result Introduces a Lipschitz value iteration method to monotonically tighten interval bounds.
This paper studies optimal approximation factors in misspecified off-policy RL, identifying key factors under various settings.
problem Understanding optimal approximation factors in misspecified off-policy value function estimation.
method Examined various settings including weighted -norm, norm, state aliasing, and state coverage.
result Established optimal asymptotic approximation factors for different norms and identified two instance-dependent factors for norm.
Paper introduces DQPOPE for estimating return distributions in reinforcement learning.
problem Estimating the entire return distribution from off-policy data.
method Deep quantile process regression for distributional off-policy evaluation.
result DQPOPE achieves statistical advantages by estimating full return distribution with same sample size.
A new estimator for evaluating policies in unknown environments.
problem Evaluating policies when both logging policy and value function are unknown.
method Doubly-Robust (DR) off-policy evaluation (OPE) estimator, DRUnknown, that estimates both the logging policy and value function.
result DRUnknown achieves the smallest asymptotic variance and is optimal when both models are correctly specified.
New estimator reduces variance in off-policy evaluation for contextual bandits.
problem High variance in current OPE methods for contextual bandits.
method Marginal Density Ratio (MR) estimator focusing on marginal distribution shift.
result MR estimator reduces variance compared to IPW and DR methods.
Study online learning with off-policy feedback in adversarial bandit problems.
problem Learning with limited direct feedback in sequential decision making.
method Proposed algorithms that adapt pessimistic reward estimators to handle unknown behavior policy.
result Guaranteed regret bounds scaling with policy mismatch, improving performance against well-covered comparators.
Study improves unbiased recommender learning by addressing missing-reward bias.
problem Data bias caused by missing-reward observations in recommender systems.
method Proposes a novel estimator using propensity scores to mitigate both position and reward bias.
result The proposed estimator outperforms other methods, even with increased reward observation bias.
New estimator improves off-policy evaluation for large action spaces.
problem Conventional importance-weighting approaches suffer from excessive variance in off-policy evaluation for large discrete action spaces.
method Proposes OffCEM estimator based on conjunct effect model (CEM), applying importance weighting only to action clusters and using model-based reward estimation for residual effects.
result Proposed estimator is unbiased under local correctness condition, providing substantial improvements in OPE especially with many actions.
Study optimizes portfolio allocation policies using off-policy data and constraints.
problem Optimizing portfolio allocation policies under constraints using off-policy data.
method Solves a minimax objective with off-policy estimators and online learning to control constraint violations.
result Constructs near-optimal allocation policies for various regimes of operation and constraints.
Proposes a method to estimate policy values in reinforcement learning with unmeasured confounders.
problem Estimating policy values in reinforcement learning with unmeasured confounders.
method Develops a two-way deconfounder algorithm using a neural tensor network to learn unmeasured confounders and system dynamics.
result Consistent policy value estimation through model-based estimator.
We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon pr…
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
New algorithm reduces bias in off-policy reinforcement learning.
problem Challenges in designing off-policy reinforcement learning algorithms.
method Doubly robust off-policy actor-critic (DR-Off-PAC) with a single timescale structure.
result Establishes the first overall sample complexity analysis for a single time-scale off-policy AC algorithm.
Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that significantly reduces the variance of infinite-horizon off-policy evaluation by estimating the stationar…
Improves RL algorithms with two techniques.
problem Enhance off-policy RL performance.
method Formulates RL as proximal point iteration; uses value functions for improved action value estimate.
result Significant performance improvement on RL benchmarks.
CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.
problem Estimating value of a target policy using only behavior policy data.
method Function space embedding, generalized empirical likelihood method, Lagrangian optimization.
result Valid confidence intervals with tighter and more accurate estimates than existing methods.
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.