FOCAL tackles offline meta-reinforcement learning with efficient task inference and behavior regularization.
problem Efficiently adapt RL algorithms to unseen tasks without interactions, addressing bootstrapping errors and robust task inference.
method FOCAL combines behavior regularization, a deterministic context encoder, and a negative-power distance metric for efficient task inference.
result FOCAL outperforms prior algorithms on meta-RL benchmarks, demonstrating computational efficiency.
Study validates metrics for offline MBO using diffusion models.
problem Evaluate metrics for offline MBO without ground truth oracle.
method Propose and quantify validation metrics over datasets.
result Identify most effective validation metrics.
The paper investigates heavy-tailed behavior in offline SGD, showing it approximates power-law tails.
problem Understanding heavy-tailed behavior in offline (multi-pass) SGD with finite data.
method Proves nonasymptotic Wasserstein convergence bounds for offline SGD to online SGD.
result Offline SGD exhibits approximate power-law tails as the number of data points increases.
A new framework for offline RL improves policy flexibility and regularity.
problem Lack of environmental interactions in offline RL leads to poor policy performance.
method Proposes a behavior-regularized implicit policy framework with modified policy-matching methods.
result The framework improves policy effectiveness and robustness beyond static datasets.
Paper addresses inconsistency between offline and online LTR performance.
problem Inconsistency between offline and online LTR performance in E-commerce.
method Proposes an evaluator-generator framework to maximize evaluator score using reinforcement learning.
result Significant improvement in Conversion Rate (CR) over existing models.
Paper analyzes sample complexity of offline MABs with KL regularization.
problem Optimizing sample complexity for offline decision-making with KL-regularized metrics.
method Sharp analysis of KL-PCB, providing upper and lower bounds.
result Characterizes sample complexity for offline MABs with KL regularization.
Unified hybrid RL algorithm improves online RL performance with offline data.
problem Improving reinforcement learning performance with limited online data.
method A unified hybrid RL algorithm combining offline and online data.
result Unified algorithm achieves state-of-the-art results in sub-optimality gap and online learning regret.
A new policy switching technique improves offline RL performance.
problem Challenges in adapting off-policy algorithms to different datasets and tasks.
method Combines off-policy RL and BC, using epistemic uncertainty for policy switching.
result Outperforms individual algorithms and state-of-the-art methods on benchmarks.
Single autoregressive model outperforms ensemble methods in offline reinforcement learning.
problem Offline reinforcement learning with limited data and model errors.
method Infer system dynamics from data and optimize policies on model rollouts, using a single autoregressive model.
result Single autoregressive model achieves better performance than ensembles on the D4RL benchmark.
Paper tackles efficient IRL in offline settings with polynomial samples and runtime.
problem Efficiently learning reward functions from expert demonstrations in offline settings.
method Adapting the pessimism principle for offline RL, achieving strong guarantees.
result Achieved efficient IRL in offline and online settings with polynomial samples and runtime.
Control Contraction Metrics (CCMs) provide a nonlinear controller design involving an offline search for a Riemannian metric and an online search for a shortest path between the current and desired trajectories. In this paper, we generalize CCMs to Finsler geometry, allowing the use of non-Riemannian metrics. We provid…
Paper tackles offline meta-reinforcement learning with a new algorithm.
problem Performing reinforcement learning on limited data from a new task.
method Meta-Actor Critic with Advantage Weighting (MACAW) algorithm.
result Achieves notable gains over prior methods on offline meta-RL benchmarks.
Paper introduces a new framework to improve sample efficiency in POMDPs learning.
problem Challenges in off-policy evaluation for POMDPs, especially with hidden states.
method Exploits the metric structure of belief space to relax coverage assumptions.
result Unified analysis technique yields tighter error bounds and sample efficiency improvements.
Algorithm balances online and offline data for linear bandits.
problem Online learning with an offline dataset in linear bandits.
method Proposes a linear bandit algorithm that uses offline data early and increasingly favors exploration as the horizon grows.
result Establishes regret bounds showing competitive performance with both purely online and offline solutions.
Algorithm learns from offline data to improve performance in target environment.
problem Learning from offline data in a target environment with unknown shifts.
method Adaptive algorithm that uses offline data to improve performance when informative.
result Algorithm provably improves performance over purely online learning when offline data are informative.
Algorithm reduces online regret by leveraging offline data in linear bandits.
problem Online regret minimization in linear bandits with offline data.
method OOPE algorithm using extended D-optimal design.
result Substantial reduction in online regret compared to prior work.
New method selects best offline RL policies from logged data.
problem Hyperparameter selection challenges offline RL.
method Offline hyperparameter selection for RL algorithms.
result Reliable ranking and selection of policies across hyperparameters.
Paper presents a reinforcement learning framework for personalized music playlist generation.
problem Misalignment between offline model objectives and online user satisfaction metrics in conventional playlist recommendation methods.
method Simulation-based reinforcement learning approach using a Deep Q-Network (DQN) modified to address large state and action spaces.
result The modified DQN (AH-DQN) policy leads to better user-satisfaction metrics compared to baseline methods during online A/B tests.
CRL improves recommendation systems by reducing distribution shift.
problem Offline metrics fail to predict online performance due to distribution shift in recommender systems.
method Proposes an information-theoretic disentanglement criterion and a variational lower bound for better generalisation under distribution shift.
result CRL variants deliver substantial online gains in listener engagement compared to baseline models.
New method tackles safe reinforcement learning from offline data.
problem Learn optimal policies from fixed data while adhering to safety constraints.
method Combines offline RL with online optimization to minimize cumulative cost.
result Proves approximate optimality of the approach under certain conditions.
GDT improves reinforcement learning by matching future state information efficiently.
problem Efficient learning of multi-task policies from trajectory data.
method Generalized Decision Transformer (GDT) for offline hindsight information matching.
result GDT enables effective offline multi-task state-marginal matching and imitation learning.
Fine-tuning RL with offline data reduces online interactions.
problem Optimizing RL with limited online interactions and offline data.
method Developed algorithm extsc{FTPedel} for MDPs with linear structure.
result Optimally reduces the number of online interactions needed.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
Study finds methods to learn multiple solutions from single task in offline RL.
problem Learning multiple solutions from a single task in offline RL.
method Proposed algorithms for offline RL.
result Empirical results show learning of multiple solutions in offline RL.
This work bridges offline RL and DRL to address distributional shift.
problem Distributional shift in offline RL due to difference in state-action visitation distributions.
method Proposes offline RL algorithms using DRL framework, characterizes sample complexity under single policy concentrability.
result Demonstrates superior performance of proposed algorithms through simulations.
FOCUS improves offline RL by incorporating causal structure into world-models.
problem Learning effective policies from historical data without interaction.
method FOCUS proposes a practical algorithm that learns and leverages causal structure in offline RL.
result FOCUS outperforms plain model-based offline RL algorithms and other causal model-based RL algorithms.
BOMS enhances offline MBRL by improving model selection with Bayesian optimization.
problem Inaccurate model selection in offline MBRL due to distribution shift.
method Proposes BOMS, an active model selection framework using Bayesian optimization.
result Improves model selection with only a small amount of online interaction.
Framework reduces contextual bandit learning to offline regression with near-optimal regret.
problem Efficient learning with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that minimizes regret with near-optimal oracle calls.
result Near-optimal regret for contextual bandits with large action spaces and O(log(T)) offline oracle calls. DO-IQS recovers optimal stopping region from expert trajectories, addressing specific challenges.
problem Recovering optimal stopping region from expert trajectories with unknown gain functions.
method Dynamics-Aware Offline Inverse Q-Learning incorporating temporal information and confidence-based oversampling.
result Demonstrated performance on real and artificial data, including optimal intervention for critical events.
Paper establishes baselines for offline RL from visual observations.
problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.
method Simple baselines and benchmarking tasks for offline RL from visual observations.
result Simple modifications to existing online RL algorithms outperform existing offline RL methods.
RORL improves offline RL robustness with conservative smoothing.
problem Distribution shift and robustness issues in offline RL.
method RORL introduces regularization and conservative smoothing for robustness.
result RORL achieves state-of-the-art performance and robustness to adversarial perturbations.
Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-p…
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
Offline RL with pre-trained features amplifies errors even under mild shifts.
problem Sample-efficient offline RL with pre-trained features under mild distribution shift.
method Empirical study of offline RL with pre-trained neural representations.
result Substantial error amplification occurs even with pre-trained features, requiring stronger conditions for successful offline RL.
A new offline RL framework unifies imitation learning and vanilla offline RL.
problem Learning from expert datasets without active data collection.
method A new offline RL framework that interpolates between imitation learning and vanilla offline RL, using a weak concentrability coefficient and a lower confidence bound algorithm.
result LCB algorithm achieves a faster rate of 1/N for nearly-expert datasets, and is adaptively optimal for the entire data composition range. Offline RL policies should adapt to unknown aspects of the environment.
problem Uncertainty in offline RL datasets leads to suboptimal policies.
method Adaptive policies that consider all transitions seen so far, solving an implicit POMDP.
result Optimal adaptive policies improve offline RL performance.
Anchor-TS uses median anchoring to improve online decision-making from offline data with distribution shift.
problem Improving online decision-making from offline data with distribution shift.
method Sample-Mean Anchored Thompson Sampling (Anchor-TS) with median anchoring.
result Anchor-TS safely leverages offline data to accelerate online learning and reduces regret.
New algorithm for offline RL with linear approx in MDPs and MGs, nearly optimal.
problem Offline RL with linear function approximation in MDPs and MGs.
method Pessimism-based algorithm with uncertainty decomposition via reference function.
result Nearly minimax optimal performance in offline RL for MDPs and MGs.
Safe offline RL for chemical reactors using input convex neural networks.
problem Safe control of exothermic polymerization reactors using historical data.
method Gymnasium-compatible simulation, behaviour cloning, implicit Q-learning, input convex neural networks (PICNNs).
result Offline RL with convex action correction outperforms traditional control approaches.
This paper bridges offline and online RL by studying policy finetuning with a reference policy.
problem Sample-efficient reinforcement learning in online and offline settings.
method Design of policy finetuning algorithms and analysis of sample complexity.
result Theoretical analysis shows that the optimal policy finetuning algorithm is either offline reduction or purely online RL.
Survey of offline RL theory and practical algorithm design challenges.
problem Optimizing return from fixed agent trajectories without additional interactions.
method Theoretical insights and practical algorithm design.
result Conditions for practical offline RL algorithms and their limitations.
A3RL combines online and offline RL with active sampling to improve policy learning.
problem Combining online and offline RL for sample efficiency and robustness.
method A3RL uses a confidence-aware Active Advantage Aligned (A3) sampling strategy to prioritize data from both online and offline sources.
result A3RL outperforms competing online RL techniques that use offline data.
A simple approach to offline RL without additional complexity.
problem Learning from a fixed dataset of actions with value estimation errors.
method Adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.
result Matches the performance of state-of-the-art offline RL algorithms with minimal changes.
New assumptions and algorithm solve offline two-player zero-sum Markov games.
problem Solving offline two-player zero-sum Markov games under insufficient assumptions.
method Proposed unilateral concentration assumption and pessimism-type algorithm.
result Algorithm efficiently learns Nash equilibrium under unilateral concentration.
DeepCO uses deep learning for offline combinatorial optimization in warehouse operations.
problem Optimizing warehouse operation sequences in offline settings.
method DeepCO framework utilizing distribution regularized optimization for TSP.
result DeepCO reduces route length by 5.7% on average for TSP problems.
New framework converts offline to online estimation using black-box offline estimators.
problem Convert offline estimation algorithms to online estimation algorithms.
method Oracle-Efficient Online Estimation (OEOE) framework.
result Achieves near-optimal online estimation error via black-box offline estimators.
PASTA optimizes assortment selection using pessimism principle.
problem Optimizing assortment selection with limited data coverage.
method Pessimistic Assortment Optimization (PASTA) based on the principle of pessimism.
result PASTA correctly identifies optimal assortment with minimal data coverage.