Survey explores how transfer learning improves deep reinforcement learning.
problem Challenges in reinforcement learning efficiency and effectiveness.
method Categorizes and analyzes transfer learning approaches.
result Transfer learning enhances reinforcement learning performance.
Paper introduces a new method for improving reinforcement learning performance using transfer learning.
problem Improving reinforcement learning performance with limited sample sizes in dynamic decision-making scenarios.
method Developed a novel ``re-weighted targeting procedure'' and ``transfer deep Q∗-learning'' approach. result Demonstrated improved reinforcement learning performance through strategic sample construction.
WAPPO optimizes feature distributions for better visual transfer in RL.
problem Improving visual transfer in reinforcement learning.
method WAPPO uses Wasserstein Confusion to minimize feature distribution distance.
result WAPPO outperforms previous methods in visual transfer across different environments.
New method automates asymmetric choice for better skill transfer in reinforcement learning.
problem Improving sample efficiency and transferability of reinforcement learning agents.
method Attentive Priors for Expressive and Transferable Skills (APES) using hierarchical KL-regularization.
result APES automates asymmetric choice, leading to better skill transfer across sequential tasks.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Researchers use DT to transfer policies from one environment to another using causal reasoning.
problem Adapting to changes in environmental dynamics in reinforcement learning.
method Applying causal counterfactual reasoning to Decision Transformer (DT) architecture for policy transfer.
result DT successfully transfers a learned policy to new environments while retaining most of the reward.
New method for RL tasks transfer using Lipschitz continuity.
problem Knowledge transfer in RL tasks over time.
method Established Lipschitz continuity between MDPs and applied it to RL.
result Improved convergence rate and no negative transfer with high probability.
PTF accelerates RL by reusing source policies without measuring task similarity.
problem Leveraging prior knowledge for faster RL.
method Adaptive Policy Transfer Framework (PTF) for RL.
result Significantly accelerates RL learning process and surpasses state-of-the-art methods.
New approach transfers rewards learned in one environment to reinforcement learning in a new environment.
problem Transfer of rewards learned using inverse reinforcement learning from one environment to a new, different environment.
method Formulate the problem as a joint system of Bellman equations, develop minimax estimators for the target soft-q-function, solve the source and target system of equations jointly. result The coupled approach removes the first-order influence of source Bellman residual error compared to the sequential approach.
Transferring knowledge across a sequence of reinforcement-learning tasks is challenging, and has a number of important applications. Though there is encouraging empirical evidence that transfer can improve performance in subsequent reinforcement-learning tasks, there has been very little theoretical analysis. In this p…
Paper analyzes AIRL in high-dimensional spaces using random matrix theory.
problem AIRL's performance challenges in high-dimensional environments.
method Examined the rank of the matrix derived from transition matrix, applied random matrix theory.
result High-dimensional scenarios reveal transfer limitations not inherent to AIRL framework.
Designs a framework to transfer causal models between similar environments.
problem Transferability of causal models between different but similar environments.
method Object-oriented representations and continuous optimization for structure learning.
result Demonstrates advantages in gridworld settings using reinforcement learning.
One question central to Reinforcement Learning is how to learn a feature representation that supports algorithm scaling and re-use of learned information from different tasks. Successor Features approach this problem by learning a feature representation that satisfies a temporal constraint. We present an implementation…
Develops HMRL for sparse reward RL problems, improving meta policy efficiency and transferability.
problem Difficulty in learning meta policies for sparse reward RL problems.
method Hyper-Meta RL framework with cross-environment meta state embedding and shaped meta reward.
result Improves meta policy generalization and efficiency for sparse reward RL problems.
Pre-trained language model boosts RL efficiency.
problem Low sample efficiency in RL, especially in lifelong learning.
method Use a pre-trained task-independent language model for transfer learning.
result Goal-conditional RL agents become more sample efficient.
In this paper we combine one method for hierarchical reinforcement learning - the options framework - with deep Q-networks (DQNs) through the use of different "option heads" on the policy network, and a supervisory network for choosing between the different options. We utilise our setup to investigate the effects of ar…
Taxonomy of knowledge modalities in RL for better transfer.
problem Tackling the transfer of knowledge in reinforcement learning.
method Systematic analysis of knowledge modalities in RL.
result A taxonomy of knowledge modalities in reinforcement learning.
Paper tackles robust knowledge transfer in parallel RL tasks.
problem Transfer knowledge from low-tier to high-tier tasks in parallel RL without shared dynamics or reward functions.
method Identifies Optimal Value Dominance condition and proposes online learning algorithms for both tasks.
result Achieves constant regret on partial states and near-optimal regret when tasks are dissimilar.
Paper proposes operator deep Q-learning for quick reward adaptation.
problem Standard RL can only handle one reward function and struggles with unseen rewards.
method Develops operator neural networks to map reward functions to value functions.
result Operator deep Q-learning can quickly adapt to new reward functions.
Exploration and adaptation to new tasks in a transfer learning setup is a central challenge in reinforcement learning. In this work, we build on the idea of modeling a distribution over policies in a Bayesian deep reinforcement learning setup to propose a transfer strategy. Recent works have shown to induce diversity i…
Agents trained with deep reinforcement learning algorithms are capable of performing highly complex tasks including locomotion in continuous environments. We investigate transferring the learning acquired in one task to a set of previously unseen tasks. Generalization and overfitting in deep reinforcement learning are …
Paper tackles action delays in reinforcement learning, proposing a delay-aware framework.
problem Action delays degrade reinforcement learning performance in real-world systems.
method Formal definition of delay-aware MDP, transformation into standard MDP with augmented states, delay-aware model-based reinforcement learning framework.
result Proposed framework is more efficient in training and transferable between systems with various delay durations.
HySRL improves RL sample efficiency with shifted-dynamics data.
problem Leveraging historical data with shifted dynamics to improve sample efficiency in RL.
method HySRL, a hybrid transfer RL algorithm that uses prior information on dynamics shift to achieve better sample complexity.
result HySRL achieves problem-dependent sample complexity and outperforms pure online RL.
MuJAM learns traffic signal control policies that generalize to unseen intersections and traffic conditions.
problem Lack of transferability in reinforcement learning methods for traffic signal control.
method Model-based graph reinforcement learning with explicit coordination and generalization to both cyclic and acyclic constraints.
result MuJAM outperforms existing methods in zero-shot and larger transfer settings.
The objective of transfer reinforcement learning is to generalize from a set of previous tasks to unseen new tasks. In this work, we focus on the transfer scenario where the dynamics among tasks are the same, but their goals differ. Although general value function (Sutton et al., 2011) has been shown to be useful for k…
Proposes a method to improve few-shot transfer in off-dynamics RL.
problem Traditional RL struggles with transferring policies between environments with different dynamics.
method Introduces a penalty to regulate source-trained policies in target environments with limited data.
result Improves performance in various off-dynamics RL scenarios compared to existing methods.
A new MARL framework for community-based cooperation with transfer and active exploration.
problem Flexible coordination patterns in multi-agent systems with community structures.
method Community-based multi-agent reinforcement learning with transfer and active exploration.
result Provably convergent actor-critic algorithms for structured information sharing and transfer learning.
This work improves transferability of rewards inferred from expert demonstrations.
problem Transferability of rewards inferred from expert demonstrations under limited access to the expert's policy.
method Proposed principal angles as a measure of similarity and dissimilarity between transition laws. Established sufficient conditions for transferability under limited access.
result Two key results on sufficient conditions for transferability to any and local changes in transition laws.
Time-variant value function transfer method for RL.
problem Transfer learning with time-variant task distributions.
method Variational approach leveraging temporal structure.
result The proposed method outperforms time-invariant approach in experiments.
Paper improves AIRL by enhancing policy imitation and addressing reward recovery issues.
problem Inadequate policy imitation and limited transferable reward recovery in AIRL.
method Substituted built-in algorithm with SAC for policy updating and proposed PPO-AIRL + SAC hybrid framework.
result SAC improves policy imitation but hinders reward recovery; PPO-AIRL + SAC achieves satisfactory transfer effect.
The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific policies or skills but almost no focus on composing necessary, task-agnostic skills to find a solution to …
In this paper, we present a new approach to Transfer Learning (TL) in Reinforcement Learning (RL) for cross-domain tasks. Many of the available techniques approach the transfer architecture as a method of speeding up the target task learning. We propose to adapt and reuse the mapped source task optimal-policy directly …
Transfer learning benefits vary in DRL, affecting representation and policy learning.
problem Understanding how much of DRL's sample complexity is due to representation learning.
method Transfer learning experiments comparing performance on related games.
result Benefits of transfer learning are highly variable and non-symmetric across tasks.
SF-DQN improves RL transfer by learning successor features.
problem Transfer RL with shared dynamics but different reward functions.
method Decomposes Q-function into SF and reward mapping; uses GPI for policy improvement.
result SF-DQN with GPI converges faster and generalizes better than traditional RL methods.
Sharing knowledge between tasks is vital for efficient learning in a multi-task setting. However, most research so far has focused on the easier case where knowledge transfer is not harmful, i.e., where knowledge from one task cannot negatively impact the performance on another task. In contrast, we present an approach…
Agent learns to trade currency pairs with improved risk management.
problem Improving systematic FX trading performance with online transfer learning.
method Online inductive transfer learning using feature representation from Gaussian mixture model to a reinforcement learning agent.
result Annualized portfolio information ratio of 0.52, compound return of 9.3%.
Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learning a universal value function (Schaul et al., 2015), which generalizes over goals and states, has previously been shown to be useful for transfer. However, successor features are b…
This work proposes a MFO framework for simultaneously evolving DQL models.
problem Complex DQL models struggle to converge to optimal policies due to exploration and sparse rewards.
method Combines meta-heuristic optimization, Transfer Learning, and DQL to automate knowledge transfer and policy learning.
result Framework automates knowledge transfer and policy learning of distributed RL agents.
AIRL learns robust, generalizable reward functions from demonstrations.
problem Learning robust reward functions from demonstrations for changing environments.
method Adversarial Inverse Reinforcement Learning (AIRL) with hierarchical disentangled rewards.
result Generalizable policies and comparable results to state-of-the-art methods.
A key question in Reinforcement Learning is which representation an agent can learn to efficiently reuse knowledge between different tasks. Recently the Successor Representation was shown to have empirical benefits for transferring knowledge between tasks with shared transition dynamics. This paper presents Model Featu…
This work shows how to use simulators to learn efficient exploration in real-world RL.
problem Sample complexity of real-world reinforcement learning.
method Coupling exploratory policies learned in simulators with practical approaches.
result Polynomial sample complexity in real world, exponential improvement over direct sim2real transfer.
Deep reinforcement learning agents have recently been successful across a variety of discrete and continuous control tasks; however, they can be slow to train and require a large number of interactions with the environment to learn a suitable policy. This is borne out by the fact that a reinforcement learning agent has…
Survey on curriculum learning for reinforcement learning.
problem Efficiently learning complex tasks in reinforcement learning.
method Framework for curriculum learning in reinforcement learning.
result Survey and classification of existing CL methods.
Transfer learning significantly accelerates the reinforcement learning process by exploiting relevant knowledge from previous experiences. The problem of optimally selecting source policies during the learning process is of great importance yet challenging. There has been little theoretical analysis of this problem. In…
We propose a probabilistic framework to directly insert prior knowledge in reinforcement learning (RL) algorithms by defining the behaviour policy as a Bayesian posterior distribution. Such a posterior combines task specific information with prior knowledge, thus allowing to achieve transfer learning across tasks. The …
Improves imitation learning in RL by learning reward function efficiently.
problem Lack of effective reward function approximation in AIRL for imitation tasks.
method Proposes Off-Policy AIRL that combines adversarial learning with efficient reward function approximation.
result Shows superior imitation performance and efficiency compared to state-of-the-art AIL algorithms.
Learning from prior tasks and transferring that experience to improve future performance is critical for building lifelong learning agents. Although results in supervised and reinforcement learning show that transfer may significantly improve the learning performance, most of the literature on transfer is focused on ba…
Deep neural networks are data hungry models and thus face difficulties when attempting to train on small text datasets. Transfer learning is a potential solution but their effectiveness in the text domain is not as explored as in areas such as image analysis. In this paper, we study the problem of transfer learning for…