The paper shows how to improve policies on- and off-policy using bounds.
problem Improving reinforcement learning policies on- and off-policy.
method Lower bounding the performance difference of two policies to ensure monotonic improvement from mixture samples.
result An optimization procedure that applies the proposed bound can be seen as an off-policy natural policy gradient method.
New algorithm BEAR reduces instability in off-policy Q-learning.
problem High sensitivity of off-policy Q-learning methods to data distribution.
method Identified and mitigated bootstrapping error through constrained action selection.
result BEAR algorithm learns robustly from various off-policy distributions.
ReF-ER improves RL by selectively updating experiences based on policy similarity.
problem Deterioration of experience replay accuracy when policies diverge.
method Enforces policy similarity in replay memory by skipping unlikely experiences and regulating policy changes.
result ReF-ER consistently improves RL performance across various benchmarks.
Enhances SAC for better sample efficiency in continuous-action tasks.
problem Improving sample efficiency in soft actor-critic algorithms.
method Integrating Emphasizing Recent Experience (ERE) with Soft Actor-Critic (SAC) and Priority Experience Replay (PER).
result ERE significantly improves sample efficiency compared to vanilla SAC, especially for continuous-action tasks.
P3O merges on-policy and off-policy updates without extra hyper-parameters.
problem Combining on-policy and off-policy RL algorithms to reduce sample complexity.
method Interleaves off-policy updates with on-policy updates using effective sample size.
result P3O reduces sample complexity of state-of-the-art algorithms.
The paper introduces a novel method for stable off-policy learning using value function chaining.
problem Stability issues in off-policy reinforcement learning.
method The approach involves learning on-policy first, then chaining off-policy value estimates.
result The method guarantees convergence and can approximate off-policy TD solutions.
A novel framework optimizes experience replay for reinforcement learning.
problem Optimizing the replay policy for better performance in off-policy reinforcement learning.
method Proposes a novel ERO framework that alternately updates the agent policy and replay policy.
result Demonstrates improved performance in various continuous control tasks.
RO-TD learns sparse value functions efficiently.
problem Learning sparse value functions efficiently.
method RO-TD integrates off-policy convergent gradient TD methods and online convex regularization.
result RO-TD learns sparse value functions with low computational complexity.
Method selects best estimator for off-policy evaluation.
problem Choosing the best estimator for off-policy evaluation.
method Generic data-driven method for estimator selection.
result Method is competitive with oracle estimator, up to a constant factor.
Meta-RL algorithm improves sample efficiency and adaptability.
problem Challenges in meta-reinforcement learning, especially on-policy experience and task uncertainty.
method Develops an off-policy meta-RL algorithm that probabilistically infers task variables.
result Significantly outperforms prior algorithms in sample efficiency and asymptotic performance.
Efficient actor-critic learning with shared experience replay improves data efficiency.
problem Challenges in actor-critic reinforcement learning with experience replay and off-policy learning stability.
method Combining actor-critic algorithms with shared experience replay, analyzing V-trace, proposing a trust region scheme.
result State-of-the-art data efficiency on Atari achieved with 200M environment frames.
Stabilizes policy optimization with off-policy data using divergence augmentation.
problem Premature convergence and instability in policy optimization with off-policy data.
method Incorporates Bregman divergence between behavior and current policies to ensure safe policy updates.
result Empirically shows better performance in data-scarce scenarios compared to other algorithms.
Paper proposes a method to create more reliable confidence intervals for off-policy evaluations.
problem Creating reliable confidence intervals for off-policy evaluations.
method Proposes a deeply-debiasing procedure to construct efficient, robust, and flexible confidence intervals.
result Validated by theoretical results and numerical experiments, the method improves the reliability of off-policy evaluations.
The paper tackles off-policy learning from multiple loggers.
problem Learning policies from multiple historical logs in real-world applications.
method Uses counterfactual estimators to learn policies from multi-logger data, analyzes generalization error, and introduces a constrained optimization problem.
result The proposed methods achieve better performance than state-of-the-arts.
New method estimates state-action stationary distribution for better off-policy policy evaluation.
problem Accurately estimating state-action stationary distribution for off-policy policy evaluation.
method Estimated Mixture Policy (EMP) for state and state-action stationary distribution corrections.
result Empirical validation shows improved accuracy over state-of-the-art methods.
Boosting for off-policy learning reduces empirical risk.
problem Learning from logged bandit feedback without labeled data.
method A boosting algorithm optimizing policy's expected reward.
result Excess empirical risk decreases with each round of boosting.
Paper introduces risk assessment for contextual bandits without experiments.
problem Evaluate policies using logged data in context bandits.
method Lipschitz risk functionals and Off-Policy Risk Assessment (OPRA) framework.
result OPRA provides finite sample guarantees for various risk estimates.
New method for off-policy evaluation in contextual bandits reduces bias-variance tradeoff.
problem Improving off-policy evaluation in contextual bandits.
method Asymptotically optimal doubly robust estimator with shrinkage of importance weights.
result New estimators outperform state-of-the-art methods in experiments.
New methods for anytime-valid off-policy inference in contextual bandits.
problem Estimating properties of hypothetical policies in adaptive experiments.
method Modern martingale techniques for comprehensive OPE inference.
result Valid anytime inference for off-policy mean reward values and entire reward distributions.
Paper addresses off-policy evaluation and learning with covariate shift.
problem Evaluating and training a new policy using historical data with a covariate shift.
method Derives efficiency bounds and proposes doubly robust estimators for OPE and OPL under covariate shift.
result Proposes estimators for off-policy evaluation and learning under covariate shift.
New method estimates off-policy data without needing known behavior policy.
problem Long-horizon reinforcement learning with limited on-policy data.
method Formulates problem as fixed point of operator, uses RKHS for estimation.
result Asymptotic consistency and finite-sample generalization proven.
DOLCE improves off-policy evaluation and learning by decomposing effects.
problem Bias in off-policy evaluation and learning due to policy mismatch.
method Uses lagged contexts and a moment-based training procedure to decompose and cancel bias.
result DOLCE achieves substantial improvements in off-policy evaluation and learning.
Concept-driven OPE reduces variance in off-policy decision evaluation.
problem High variance in off-policy decision evaluation due to limited sample sizes.
method Integrating human-explainable concepts into OPE to reduce variance.
result Concept-based OPE estimators remain unbiased and reduce variance when concepts are known and predefined.
A new resampling strategy, Importance Resampling, improves sample efficiency and reduces variance in off-policy prediction.
problem High variance updates in importance sampling for off-policy prediction.
method Importance Resampling (IR) resamples experience from a replay buffer and applies standard on-policy updates, avoiding importance sampling ratios.
result Importance Resampling (IR) shows improved sample efficiency and lower variance updates compared to other methods.
LiDER refreshes past experiences in RL by dreaming about them.
problem Improving data efficiency in off-policy RL algorithms.
method Refreshing past experiences in a replay buffer using the current policy.
result LiDER consistently improves performance in Atari games.
New method for rigorous confidence intervals in off-policy evaluation.
problem Evaluate new policies from off-policy data without executing them.
method Variational framework using kernel Bellman loss.
result Efficient method for tight confidence intervals in various settings.
Study online learning with off-policy feedback in adversarial bandit problems.
problem Learning with limited direct feedback in sequential decision making.
method Proposed algorithms that adapt pessimistic reward estimators to handle unknown behavior policy.
result Guaranteed regret bounds scaling with policy mismatch, improving performance against well-covered comparators.
CF-GPS learns policies from logged data by considering counterfactual outcomes.
problem Learning policies from limited real experience in complex environments.
method Assumes logged real experience and models counterfactual outcomes. Uses structural causal models for evaluation.
result Improves policy evaluation and search results on a grid-world task.
New insights into experience replay in RL algorithms.
problem Understanding the impact of replay capacity and replay ratio in Q-learning.
method Systematic and extensive analysis of experience replay in Q-learning methods, focusing on replay capacity and replay ratio.
result Greater replay capacity significantly improves performance for certain algorithms, while other techniques offer limited benefit.
Develops RL algorithm for lifelong non-stationary environments.
problem Challenges of reinforcement learning in environments with persistent change.
method Formalizes lifelong non-stationarity, uses latent variable models, and leverages online learning and probabilistic inference.
result Substantial improvement in performance over non-reasoning approaches in lifelong non-stationary environments.
Paper introduces control variate to reduce variance in off-policy reinforcement learning.
problem High variance in off-policy reinforcement learning.
method Introduces control variate technique to Expected Sarsa(λ) and proposes ES(λ)-CV algorithm.
result Proposed ES(λ)-CV algorithm has lower variance than Expected Sarsa(λ).
Proposes a method to estimate policy values in reinforcement learning with unmeasured confounders.
problem Estimating policy values in reinforcement learning with unmeasured confounders.
method Develops a two-way deconfounder algorithm using a neural tensor network to learn unmeasured confounders and system dynamics.
result Consistent policy value estimation through model-based estimator.
Paper finds efficient OPE estimator for multiple logging policies with minimum variance.
problem Finding optimal importance sampling weights for multiple logging policies with varying variances.
method Established efficiency bound under stratified sampling and proposed an estimator achieving this bound.
result Proposed estimator achieves minimum variance for any instance.
MetaGenRL learns a general objective function from diverse agents.
problem Generalizing to new environments in reinforcement learning.
method MetaGenRL distills experiences from many agents into a low-complexity neural objective function.
result MetaGenRL can generalize to new environments and outperforms human-engineered algorithms.
Study improves unbiased recommender learning by addressing missing-reward bias.
problem Data bias caused by missing-reward observations in recommender systems.
method Proposes a novel estimator using propensity scores to mitigate both position and reward bias.
result The proposed estimator outperforms other methods, even with increased reward observation bias.
New methods improve off-policy evaluation for survival outcomes with censoring.
problem Systematic underestimation of policy performance due to censoring bias in survival outcomes.
method Proposes IPCW-IPS and IPCW-DR to handle censoring bias in survival outcomes.
result The proposed methods are unbiased and achieve double robustness.
Paper reformulates policy gradient for on/off-policy reinforcement learning.
problem Combining on/off-policy methods for real-world reinforcement learning.
method Reformulates policy gradient into a perceptron-like loss function, removing on/off policy distinction.
result New formulation enables off-policy training with arbitrary policies.
AWR simplifies RL with simple supervised learning.
problem Developing scalable RL algorithms for off-policy data.
method Two supervised learning steps: value function and policy.
result AWR achieves competitive performance and effective policies.
New approach for off-policy learning in contextual bandits with performance guarantees.
problem Improving performance of logging policies in contextual bandits.
method PAC-Bayesian analysis of policy mixtures, providing tighter generalization bounds and tractable optimization algorithms.
result Proved tighter generalization bounds and demonstrated effectiveness in practical scenarios.
Paper proposes a method to improve off-policy reinforcement learning in batch settings.
problem Challenges in applying off-policy reinforcement learning to batch data.
method Uses a learned prior, the advantage-weighted behavior model (ABM), to bias RL policies.
result Improves performance on various RL tasks, including robot control.
Improved off-policy evaluation for MDPs with weak distributional overlap.
problem Evaluation of policies when target and data-collection distributions are not strongly overlapping.
method Truncated Doubly Robust (TDR) estimators for off-policy evaluation in MDPs under weak distributional overlap.
result TDR estimators can recover large-sample behavior and are consistent even when distribution ratios are not square-integrable.
CoinDICE estimates confidence intervals for unknown behavior policies in reinforcement learning.
problem Estimating value of a target policy using only behavior policy data.
method Function space embedding, generalized empirical likelihood method, Lagrangian optimization.
result Valid confidence intervals with tighter and more accurate estimates than existing methods.
Study off-policy evaluation in partially observable environments, reducing bias and errors.
problem Bias and large errors in off-policy evaluation for partially observable environments.
method Defined and solved off-policy evaluation for POMDPs, introduced Decoupled POMDP model.
result Demonstrated and compared off-policy evaluation methods, showing benefits of new approach.
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.
A streamlined DRL algorithm improves sample efficiency without entropy maximization.
problem Improving sample efficiency in off-policy DRL algorithms.
method Output normalization and non-uniform sampling.
result Proposed algorithm matches SAC's performance without entropy maximization and improves sample efficiency.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
HIRO learns complex behaviors from few interactions.
problem Developing efficient hierarchical reinforcement learning methods.
method HIRO uses off-policy experience and automatic goal learning to generalize and be sample-efficient.
result HIRO learns complex behaviors from a few million samples, equivalent to a few days of real-time interaction.
Develops an off-policy method for improving chatbots with noisy rewards.
problem Improving chatbots with noisy and expensive rewards.
method Batch policy gradient method (BPG).
result Demonstrates the efficacy of BPG through synthetic and real-world experiments.