New methods for evaluating and optimizing policies in offline RL with unobserved confounders.
problem Evaluating and optimizing policies in the presence of unobserved confounders.
method Characterized settings and algorithms for consistent value estimates and lower bounds, with sample complexity guarantees.
result Proved local convergence guarantees for offline policy improvement.
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…
This work characterizes conditions for offline policy evaluation in reinforcement learning.
problem Understanding when classical methods succeed in offline policy evaluation for linear function approximation.
method Control-theoretic and linear-algebraic conditions for classical methods (FQI and LSTD).
result A precise hierarchy of regimes under which these estimators succeed, and a complete picture of their behavior.
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…
Paper addresses offline policy evaluation in RL, achieving near-optimal bounds for various policy classes.
problem Evaluate all policies in a class simultaneously for offline RL.
method Uniform convergence in OPE for various policy classes, achieving optimal episode complexity.
result Achieves optimal episode complexity of O(H^3/d_mε^2) for identifying ε-optimal policies.
Paper tackles offline SSP with value iteration for policy evaluation and learning.
problem Goal-oriented RL with offline data and cost minimization.
method Simple value iteration algorithms for OPE and offline policy learning.
result Strong instance-dependent bounds implying near-minimax optimal worst-case bounds.
New offline RL study shows exponential sample requirement for accurate policy evaluation.
problem Understanding statistical limits of offline RL with linear function approximation.
method Analyzes necessary representational and distributional conditions for sample-efficient offline reinforcement learning.
result Even with realizability and good feature coverage, offline RL requires exponential samples for accurate policy evaluation.
OPERA blends multiple OPE estimators to evaluate new policies offline.
problem Lack of reliable offline policy evaluation methods for new policies.
method Adaptive blending of multiple OPE estimators without explicit selection.
result Consistent and reliable policy evaluation framework for offline RL.
Paper establishes baselines for offline RL from visual observations.
problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.
method Simple baselines and benchmarking tasks for offline RL from visual observations.
result Simple modifications to existing online RL algorithms outperform existing offline RL methods.
Paper tackles efficient evaluation of natural stochastic policies in offline RL.
problem Efficiency issues in evaluating natural stochastic policies due to unknown evaluation policy.
method Derive efficiency bounds for tilting and modified treatment policies, propose nonparametric estimators.
result Proposed estimators attain efficiency bounds under lax conditions and enjoy partial double robustness.
New method selects best offline RL policies from logged data.
problem Hyperparameter selection challenges offline RL.
method Offline hyperparameter selection for RL algorithms.
result Reliable ranking and selection of policies across hyperparameters.
Paper addresses inconsistency between offline and online LTR performance.
problem Inconsistency between offline and online LTR performance in E-commerce.
method Proposes an evaluator-generator framework to maximize evaluator score using reinforcement learning.
result Significant improvement in Conversion Rate (CR) over existing models.
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
BCRL learns a Bellman complete representation for offline RL policy evaluation.
problem Learning a Q-function efficiently from offline data.
method BCRL learns a linear Bellman complete representation directly from data, enabling efficient OPE.
result BCRL achieves competitive OPE error and outperforms FQE in certain scenarios.
New insights on offline RL with state aggregation and trajectory data.
problem Understanding sample complexity in offline policy evaluation.
method Analyzing concentrability coefficient in aggregated Markov Transition Model.
result Sample complexity depends on concentrability coefficient in aggregated model.
Contextual bandit algorithms have become popular for online recommendation systems such as Digg, Yahoo! Buzz, and news recommendation in general. \emph{Offline} evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "p…
Develops a method to evaluate OPE robustness to hyperparameters and policies.
problem Difficulty in selecting and tuning OPE estimators due to limited experimental evaluations.
method Introduces IEOE (Interpretable Evaluation for Offline Evaluation) to assess robustness.
result Demonstrates improved evaluation of OPE estimators' reliability.
The paper shows how ignoring temporal context in recommender systems evaluation leads to false confidence, proposing a method to embed temporal context.
problem The discrepancy between offline and online recommender system performance evaluation.
method Proposes a training procedure to embed temporal context into recommender systems and validates its advantage using multi-objective optimization.
result Including temporal context in recommender systems evaluation can improve recall@20 by up to 20%.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Offline RL with pre-trained features amplifies errors even under mild shifts.
problem Sample-efficient offline RL with pre-trained features under mild distribution shift.
method Empirical study of offline RL with pre-trained neural representations.
result Substantial error amplification occurs even with pre-trained features, requiring stronger conditions for successful offline RL.
Paper tackles policy selection with logged data and limited online interactions.
problem Safe evaluation and deployment of offline reinforcement learning policies.
method Active offline policy selection combining logged data with online interaction.
result Improves upon state-of-the-art OPE estimates and pure online policy evaluation.
Normalizing flow regression approximates posterior distributions without additional sampling.
problem Bayesian inference with computationally expensive likelihood evaluations.
method Normalizing flow regression (NFR) for offline inference.
result NFR yields a tractable posterior approximation through regression on existing log-density evaluations.
Recommendation systems have been integrated into the majority of large online systems. They tailor those systems to individual users by filtering and ranking information according to user profiles. This adaptation process influences the way users interact with the system and, as a consequence, increases the difficulty …
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
RL Unplugged benchmarks offline RL methods across diverse domains.
problem Evaluate offline reinforcement learning methods without online data collection.
method Proposes a benchmark suite with diverse datasets and detailed evaluation protocols.
result Demonstrates the effectiveness of offline RL methods across various domains.
New method tackles safe reinforcement learning from offline data.
problem Learn optimal policies from fixed data while adhering to safety constraints.
method Combines offline RL with online optimization to minimize cumulative cost.
result Proves approximate optimality of the approach under certain conditions.
This paper explores how IV methods can improve Q-function estimates in offline policy evaluation.
problem Confounding in estimating Q-function using reinforcement learning.
method Integrates IV techniques into offline policy evaluation (OPE) to improve Q-function estimates.
result State-of-the-art OPE methods are closely matched in performance by some IV methods.
New method efficiently evaluates policies using trajectory data.
problem Statistically efficient policy evaluation with limited data.
method Trajectory-based approach for policy evaluation.
result Improved sample complexity for policy evaluation.
DeepCO uses deep learning for offline combinatorial optimization in warehouse operations.
problem Optimizing warehouse operation sequences in offline settings.
method DeepCO framework utilizing distribution regularized optimization for TSP.
result DeepCO reduces route length by 5.7% on average for TSP problems.
The paper proposes autoregressive models for better offline RL.
problem Offline RL policy evaluation and optimization challenges.
method Autoregressive dynamics models for sequential state and reward prediction.
result Autoregressive models outperform standard methods in log-likelihood and RL tasks.
Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are expensive in model learning. Offline evaluation methods, such as importance sampling, can alleviate s…
New OPE estimator improves offline policy evaluation for large action spaces.
problem Existing OPE estimators fail with large action spaces, leading to extreme bias and variance.
method Proposes a new estimator using marginalized importance weights and action embeddings.
result Empirical performance improvement enables reliable OPE even with many actions.
PyCFRL helps ensure fair reinforcement learning policies from offline data.
problem Ensuring fairness in reinforcement learning policies for disadvantaged groups.
method Sequential data preprocessing to learn counterfactually fair policies.
result PyCFRL implements a novel algorithm for fair RL policy learning.
Develops a support-aware framework for reserve-policy selection in advertising markets.
problem Log-based reserve-price evaluation risks weak support and subgroup harm.
method Support-aware offline decision framework converting logged evidence into certified policies.
result Preserves the best gate-passing policy while eliminating only policies with certified regret.
Offline RL policies should adapt to unknown aspects of the environment.
problem Uncertainty in offline RL datasets leads to suboptimal policies.
method Adaptive policies that consider all transitions seen so far, solving an implicit POMDP.
result Optimal adaptive policies improve offline RL performance.
BOMS enhances offline MBRL by improving model selection with Bayesian optimization.
problem Inaccurate model selection in offline MBRL due to distribution shift.
method Proposes BOMS, an active model selection framework using Bayesian optimization.
result Improves model selection with only a small amount of online interaction.
Paper tackles robust offline RL with heavy-tailed rewards.
problem Real-world applications often encounter heavy-tailed rewards, challenging offline RL.
method Proposes ROAM and ROOM algorithms using median-of-means method for robust off-policy evaluation and OPO.
result Demonstrates superior performance on heavy-tailed reward datasets compared to existing methods.
Study validates metrics for offline MBO using diffusion models.
problem Evaluate metrics for offline MBO without ground truth oracle.
method Propose and quantify validation metrics over datasets.
result Identify most effective validation metrics.
One-step policy improvement outperforms iterative RL methods on D4RL.
problem Improving offline RL without off-policy evaluation.
method One-step constrained/regularized policy improvement using on-policy Q estimates.
result One-step algorithm outperforms iterative algorithms on D4RL benchmark.
A new method combines online and offline learning to tackle contextual bandits with missing action support.
problem Learning optimal policies with logged data when the logging policy has deficient support.
method Hybrid approach using online exploration to exploit supported actions and offline learning to avoid unnecessary explorations.
result Determines an optimal policy with theoretical guarantees using minimal online explorations.
Study human-machine interaction with private info using offline RL.
problem Confounding bias and distributional mismatch in offline RL for human-guided interaction.
method Developed a novel identification result and OPE method to address confounding bias, and used pessimism to tackle distributional mismatch.
result Policy pair converges to optimal one at satisfactory rate under mild assumptions.
Benchmarking off-policy evaluation methods for complex policies.
problem Lack of comprehensive benchmarks for off-policy evaluation methods.
method Collection of challenging high-dimensional control tasks and datasets.
result Standardized measure of progress for OPE methods.
Investigates offline RL in factorisable action spaces, overcoming overestimation bias.
problem Overestimation bias in value estimates for unseen state-action pairs.
method Value-decomposition approach in DecQN, adapted for factorised discrete action spaces.
result Demonstrates the effectiveness of factorised approach in offline RL.
Safe offline RL for chemical reactors using input convex neural networks.
problem Safe control of exothermic polymerization reactors using historical data.
method Gymnasium-compatible simulation, behaviour cloning, implicit Q-learning, input convex neural networks (PICNNs).
result Offline RL with convex action correction outperforms traditional control approaches.
In academic literature, recommender systems are often evaluated on the task of next-item prediction. The procedure aims to give an answer to the question: "Given the natural sequence of user-item interactions up to time t, can we predict which item the user will interact with at time t+1?". Evaluation results obtained …
In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or…
The (contextual) multi-armed bandit problem (MAB) provides a formalization of sequential decision-making which has many applications. However, validly evaluating MAB policies is challenging; we either resort to simulations which inherently include debatable assumptions, or we resort to expensive field trials. Recently …
AHS framework selects hyperparameters for FQE with error guarantees.
problem Hyperparameter selection for FQE is challenging and affects its utility.
method AHS framework defines optimality criteria without hyperparameters.
result Error bounds match empirical observations.