Paper tackles offline meta-reinforcement learning with a new algorithm.
problem Performing reinforcement learning on limited data from a new task.
method Meta-Actor Critic with Advantage Weighting (MACAW) algorithm.
result Achieves notable gains over prior methods on offline meta-RL benchmarks.
FOCAL tackles offline meta-reinforcement learning with efficient task inference and behavior regularization.
problem Efficiently adapt RL algorithms to unseen tasks without interactions, addressing bootstrapping errors and robust task inference.
method FOCAL combines behavior regularization, a deterministic context encoder, and a negative-power distance metric for efficient task inference.
result FOCAL outperforms prior algorithms on meta-RL benchmarks, demonstrating computational efficiency.
Meta-agent learns effective exploration from offline data.
problem Design a meta-agent to quickly maximize reward in unseen tasks.
method Bayesian RL approach with adaptive neural belief estimate.
result Meta-agent learns effective exploration behavior from diverse tasks.
CausalCOMRL improves RL task representations by integrating causal relationships, enhancing generalizability.
problem Spurious correlations in context-based offline meta-reinforcement learning.
method CausalCOMRL integrates causal representation learning to uncover and incorporate causal relationships among task components.
result CausalCOMRL achieves better performance on meta-reinforcement learning benchmarks.
Meta-KeL learns kernels from offline data to improve sequential decision-making.
problem Adaptive confidence sets for prediction functions in sequential decision-making tasks.
method Meta-KeL: meta-learning a kernel from offline data; structured sparsity estimator for unknown kernel combinations.
result Valid confidence sets that become as tight as those given the true unknown kernel with increasing offline data.
Meta-gradient D4PG optimizes performance and constraint adherence in RL.
problem Balancing performance and adherence to complex constraints in RL.
method Uses meta-gradients to find a balance between expected return and minimizing constraint violations.
result Meta-gradient D4PG consistently outperforms baselines across MuJoCo domains.
ContraBAR uses contrastive learning to learn Bayes-optimal policies in RL.
problem Learning optimal policies for unknown tasks sampled from a known distribution.
method Proposes ContraBAR, a meta RL algorithm using contrastive predictive coding (CPC) for belief inference.
result ContraBAR achieves comparable performance to state-of-the-art methods and is computationally efficient.
Study tackles RLHF with diverse human feedback, showing limitations and proposing a meta-learning approach.
problem Traditional RLHF fails to balance diverse human preferences.
method Integrates meta-learning and multiple social welfare functions to optimize diverse preferences.
result Establishes sample complexity bounds for optimizing diverse social welfare functions.
Pipeline selects best RL policy from limited data.
problem Selecting best RL policy from small datasets.
method Task- and method-agnostic pipeline using multiple data splits.
result Pipeline produces higher-performing policies.
New algorithm uses density ratios for efficient online reinforcement learning.
problem Challenges in collecting exploratory data for online reinforcement learning.
method Density ratio modeling for online exploration, combining truncation and optimism.
result Sample-efficient online exploration achieved with GLOW and HyGLOW.
Safe RL with binary feedback using SABRE algorithm.
problem Safe reinforcement learning with binary safety feedback.
method SABRE algorithm, combining active learning and reinforcement learning.
result Provable safe policy with high probability, no unsafe actions during training.
Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depe…
Develops HMRL for sparse reward RL problems, improving meta policy efficiency and transferability.
problem Difficulty in learning meta policies for sparse reward RL problems.
method Hyper-Meta RL framework with cross-environment meta state embedding and shaped meta reward.
result Improves meta policy generalization and efficiency for sparse reward RL problems.
Study OOD generalization in meta-reinforcement learning using information theory.
problem Understanding how meta-reinforcement learning handles distribution shifts.
method Information-theoretic analysis of Markov Decision Processes and gradient-based algorithms.
result Established fine-grained generalization bounds for meta-reinforcement learning.
Proposes efficient user cold start recommendation via meta parameter partition.
problem User cold start in recommendation systems.
method Divides model parameters into fixed and adaptive parts, learning them separately offline and online.
result Significant improvement in AUC (2.48% absolute improvement).
FOCUS improves offline RL by incorporating causal structure into world-models.
problem Learning effective policies from historical data without interaction.
method FOCUS proposes a practical algorithm that learns and leverages causal structure in offline RL.
result FOCUS outperforms plain model-based offline RL algorithms and other causal model-based RL algorithms.
RORL improves offline RL robustness with conservative smoothing.
problem Distribution shift and robustness issues in offline RL.
method RORL introduces regularization and conservative smoothing for robustness.
result RORL achieves state-of-the-art performance and robustness to adversarial perturbations.
New method tackles safe reinforcement learning from offline data.
problem Learn optimal policies from fixed data while adhering to safety constraints.
method Combines offline RL with online optimization to minimize cumulative cost.
result Proves approximate optimality of the approach under certain conditions.
Paper establishes baselines for offline RL from visual observations.
problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.
method Simple baselines and benchmarking tasks for offline RL from visual observations.
result Simple modifications to existing online RL algorithms outperform existing offline RL methods.
This tutorial reviews offline RL methods and challenges in deep learning.
problem Extracting optimal policies from large datasets without online data.
method Review of existing offline RL algorithms and potential solutions.
result Challenges and open problems in modern deep RL methods.
Study on meta-reinforcement learning generalization in high-dimensional tasks.
problem Generalization performance of meta-reinforcement learning algorithms in high-dimensional tasks.
method High-dimensional, procedurally generated environments.
result Meta-reinforcement learning algorithms exhibit strong overfitting on challenging tasks.
Enhances reinforcement learning from sparse data.
problem Limited data for offline reinforcement learning.
method Trajectory-based data augmentation.
result Improves reinforcement learning performance.
Algorithm learns from offline data to improve performance in target environment.
problem Learning from offline data in a target environment with unknown shifts.
method Adaptive algorithm that uses offline data to improve performance when informative.
result Algorithm provably improves performance over purely online learning when offline data are informative.
Study finds methods to learn multiple solutions from single task in offline RL.
problem Learning multiple solutions from a single task in offline RL.
method Proposed algorithms for offline RL.
result Empirical results show learning of multiple solutions in offline RL.
New method learns credit prices offline without interaction.
problem Dynamic pricing of consumer credit.
method Offline deep reinforcement learning with Q-Learning.
result Effective personalized pricing policy learned without online interaction.
New method learns policies from offline data using operator models.
problem Limited understanding of approximation errors in offline reinforcement learning.
method Linking reinforcement learning to Hamilton-Jacobi-Bellman equation, proposing operator-theoretic algorithm.
result Global convergence of the value function and finite-sample guarantees derived.
New symmetries improve meta-reinforcement learning's generalization.
problem Black-box meta reinforcement learning struggles with generalization.
method Introduce symmetries to a black-box meta RL system.
result Incorporating symmetries improves generalization to new environments.
Efficient offline reinforcement learning with neural networks using differentiable function approximation.
problem Statistical efficiency of offline reinforcement learning with function approximators.
method Pessimistic fitted Q-learning (PFQL) and differentiable function approximation.
result Provably efficient offline reinforcement learning with differentiable function approximation.
New method selects best offline RL policies from logged data.
problem Hyperparameter selection challenges offline RL.
method Offline hyperparameter selection for RL algorithms.
result Reliable ranking and selection of policies across hyperparameters.
Gradient-based meta-RL fails with incorrect task distributions, leading to instability and poor performance.
problem Gradient-based meta-RL's sensitivity to task distributions causes instability and poor performance.
method Proposes meta Active Domain Randomization (meta-ADR) to learn task distributions for gradient-based meta-RL.
result Meta-ADR improves stability and generalization of MAML on simulated locomotion and navigation tasks.
Study tight offline learning bounds for linear MDPs using variance information.
problem Understanding statistical limits with linear function representations in offline reinforcement learning.
method Variance-aware pessimistic value iteration (VAPVI) that reweights Bellman residuals based on estimated variances.
result Improved offline learning bounds expressed in terms of system quantities.
Optimizes mobile notifications for multiple objectives using reinforcement learning.
problem Optimizing mobile notification systems for multiple objectives.
method End-to-end offline reinforcement learning with Double Deep Q-network and Conservative Q-learning.
result Demonstrates improved performance and benefits of the proposed approach.
Meta-reinforcement learning improves fault-adaptive control efficiency.
problem Adaptive control under abrupt system faults with strict time constraints.
method Model-agnostic meta learning (MAML) with a fault library of prior policies.
result Improved sample efficiency and quick adaptation to new faults.
Proves efficient learning of hierarchical structure in meta-reinforcement learning.
problem Lack of provable guarantees for learning hierarchical structures in reinforcement learning.
method Analyzed HRL in meta-RL setting with tabular transition dynamics, providing diversity conditions and regret bounds.
result Sample-efficient recovery of natural hierarchical structure with provable guarantees.
A simple approach to offline RL without additional complexity.
problem Learning from a fixed dataset of actions with value estimation errors.
method Adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.
result Matches the performance of state-of-the-art offline RL algorithms with minimal changes.
ORIL learns a reward function from unlabeled data to improve robot learning.
problem Leveraging unlabeled data for robot learning.
method ORIL learns a reward function from demonstrator and unlabeled trajectories, annotates data, and trains an agent via offline reinforcement learning.
result ORIL consistently outperforms BC agents on various robotic tasks.
Bayesian meta-reinforcement learning improves over point estimates with Laplace approximation.
problem Improving meta-reinforcement learning by providing full posterior distributions.
method Augmenting point estimates with Laplace approximation for full posterior distributions.
result Our method performs similarly to variational baselines with fewer parameters.
New method improves meta-reinforcement learning efficiency.
problem Sample inefficiency in meta-reinforcement learning.
method Hindsight Foresight Relabeling (HFR) method.
result HFR improves performance on various meta-reinforcement learning tasks.
CODA resolves coordination issues in offline multi-agent reinforcement learning.
problem Coordination failure in offline multi-agent reinforcement learning.
method Diffusion-based multi-agent trajectory generator for data augmentation.
result CODA resolves coordination pathologies in continuous polynomial games and complex benchmarks.
A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.
problem Improving offline reinforcement learning performance.
method Integrates count-based conservatism into model-based offline reinforcement learning.
result The learned policy is near-optimal and outperforms existing methods.
Offline RL struggles with sample efficiency due to fundamental barriers.
problem Sample efficiency in offline RL with value function approximation.
method Analyzes the necessity of distributional and representational assumptions.
result Even with concentrability and realizability, sample complexity is polynomial in state space size.
Offline RL algorithms protect privacy while learning from sensitive data.
problem Learning sensitive data in financial, legal, and healthcare applications.
method Differential privacy guarantees for offline RL algorithms.
result Provable prevention of privacy risks with strong learning bounds.
Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement…
Algorithm learns optimal dynamic mechanisms from data.
problem Designing optimal mechanisms for dynamic settings with unknown reward functions.
method Offline reinforcement learning with pessimism principle.
result Learned mechanisms are efficient, individually rational, and truthful.
Novel Bayesian meta-reinforcement learning framework improves traffic signal control robustness.
problem Lack of robustness and stability in adaptation for traffic signal control.
method Value-based Bayesian meta-reinforcement learning framework BM-DQN with fast-adaptation variation and DQN fast-update advantage.
result Framework adapts more quickly and robustly to new scenarios than previous methods.
Pessimistic Q-learning improves sample efficiency in offline reinforcement learning.
problem Insufficient coverage and sample scarcity in offline reinforcement learning datasets.
method Pessimistic Q-learning algorithm for offline reinforcement learning, focusing on variance reduction.
result Near-optimal sample complexity achieved with the proposed algorithm.
Investigates offline RL in factorisable action spaces, overcoming overestimation bias.
problem Overestimation bias in value estimates for unseen state-action pairs.
method Value-decomposition approach in DecQN, adapted for factorised discrete action spaces.
result Demonstrates the effectiveness of factorised approach in offline RL.
Critic-regularized regression improves offline RL performance.
problem Poor performance of off-policy algorithms in offline RL.
method Critic-regularized regression (CRR) for policy learning from fixed datasets.
result CRR outperforms state-of-the-art offline RL algorithms significantly.