Proves EM algorithm guarantees for hierarchical imitation learning.
problem Learning hierarchical policies from expert demonstrations.
method Converts hierarchical imitation learning to latent variable model inference, analyzes EM algorithm.
result Proves EM algorithm converges to true parameter under regular conditions.
Novel online algorithm for hierarchical imitation learning.
problem Scalability issue in reinforcement learning and options discovery.
method Online Baum-Welch algorithm for hierarchical imitation learning.
result The online algorithm outperforms the batch version in both discrete and continuous environments.
Autonomous agents can learn by imitating teacher demonstrations of the intended behavior. Hierarchical control policies are ubiquitously useful for such learning, having the potential to break down structured tasks into simpler sub-tasks, thereby improving data efficiency and generalization. In this paper, we propose a…
The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. …
We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the…
Hierarchical decoupling improves sample efficiency for complex robots.
problem Learning long-range behaviors on complex robots.
method Two-part policy: low-level imitation and high-level transfer, with KL regularization.
result Hierarchical transfer significantly improves zero-shot high-level transfer and stabilizes learning.
Survey of methods for learning from observation without requiring expert actions.
problem Lack of access to expert actions in imitation learning.
method Survey and classification of state-only imitation learning (SOIL) methods.
result Identification of open problems and future research directions.
New algorithm trains neural nets on simple skills to learn complex tasks faster.
problem Learning complex tasks through simple imitation.
method Train neural networks on simple, easy-to-learn skills to accelerate learning of complex, hard-to-learn tasks.
result Consistently outperforms state-of-the-art baseline in training speed and performance.
We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learni…
H-ReIL learns to drive safely in near-accident scenarios.
problem Driving safely in high-risk near-accident situations.
method Hierarchical RL and IL approach.
result High-level policy switches between low-level policies for safe driving.
We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective. To accomplish this goal, our agent should not only learn to understand the intent of the demonstr…
New model shows hierarchical proof structure helps theorem provers.
problem How to efficiently learn proofs from teacher traces.
method Model proof search as MDP, analyze imitation learning from traces.
result Hierarchical proof structure leads to more efficient learning.
We introduce Compositional Imitation Learning and Execution (CompILE): a framework for learning reusable, variable-length segments of hierarchically-structured behavior from demonstration data. CompILE uses a novel unsupervised, fully-differentiable sequence segmentation module to learn latent encodings of sequential d…
Robot learns multiple tasks hierarchically by transferring knowledge.
problem Learning multiple complex tasks in open-ended environments.
method Task-oriented procedures, goal-babbling, imitation learning, active learning, intrinsic motivation.
result Robots can learn complex tasks more efficiently by transferring knowledge from simpler ones.
We present an active learning architecture that allows a robot to actively learn which data collection strategy is most efficient for acquiring motor skills to achieve multiple outcomes, and generalise over its experience to achieve new outcomes. The robot explores its environment both via interactive learning and goal…
MineRL Competition reduced reinforcement learning sample needs.
problem Sample inefficiency in reinforcement learning.
method Human demonstrations and imitation learning integrated into reinforcement learning algorithms.
result Top solutions used deep reinforcement learning and imitation learning.
Consider an imitation learning problem that the imitator and the expert have different dynamics models. Most of the current imitation learning methods fail because they focus on imitating actions. We propose a novel state alignment-based imitation learning method to train the imitator to follow the state sequences in e…
ADVISOR dynamically balances imitation and reinforcement learning to overcome the imitation gap.
problem The gap between imitation learning and reinforcement learning when teaching agents have privileged information.
method Adaptive Insubordination (ADVISOR) dynamically weights imitation and reward-based reinforcement learning losses.
result On-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, and their combinations.
Compared to reinforcement learning, imitation learning (IL) is a powerful paradigm for training agents to learn control policies efficiently from expert demonstrations. However, in most cases, obtaining demonstration data is costly and laborious, which poses a significant challenge in some scenarios. A promising altern…
Reinforcement learning mimics expert behavior.
problem Learning from expert demonstrations in reinforcement learning.
method Reduction to reinforcement learning with a stationary reward.
result Expert reward can be recovered and imitation learning is bounded.
A new benchmark task for evaluating policy learning in complex, high-dimensional action spaces.
problem Lack of a commonly accepted benchmark for evaluating policy learning in hierarchical tasks with high-dimensional action spaces.
method Proposed DinerDash Gym benchmark and Decomposed Policy Graph Modelling (DPGM) algorithm.
result DPGM achieves significant improvement over baselines and effectively injects domain knowledge.
Guarantees for third-person imitation learning from offline data.
problem Improving generalizability in imitation learning.
method Problem-dependent statistical learning guarantees for third-person imitation from offline observation.
result Strong performance guarantees for transferred policies in the offline setting.
New metric solves correspondence problem for robotic arm imitation learning.
problem Establishing corresponding states and actions between different robotic arms.
method Introducing a distance measure between dissimilar robotic arms and using it as a loss function.
result The distance measure effectively learns imitation policies by minimizing distance between robotic arms.
New method uses bi-level optimization to learn useful representations for imitation learning.
problem Learning useful representations for multiple tasks in imitation learning settings.
method Formulates representation learning as a bi-level optimization problem.
result Bi-level optimization framework provides sample complexity benefits for imitation learning.
Researchers improved Minecraft game performance using imitation learning.
problem Achieving state-of-the-art performance in immersive environments like Minecraft.
method Applied imitation learning to Minecraft, optimizing network architecture, loss function, and data augmentation.
result Reported stronger results than previous experiments, reaching second place in a competition.
New self-imitation learning method improves performance in continuous control tasks.
problem Improving off-policy learning in continuous control tasks.
method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.
result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.
Paper tackles robust imitation learning from noisy demonstrations.
problem Learning from noisy demonstrations is challenging.
method Optimizes a classification risk with a symmetric loss, combining pseudo-labeling and co-training.
result Our method is more robust than state-of-the-art methods.
Paper uses Sinkhorn distances to improve imitation learning effectiveness.
problem Improving imitation learning algorithms by comparing occupancy measures.
method Formulates imitation learning as Sinkhorn distance minimization, combining optimal transport and cosine distances.
result Proposes a new critic network and transport plan that guide imitation learning.
Proof shows imitation of expert's reward and solutions in multi-objective optimization.
problem Multi-objective optimization with reward and solution imitation.
method Wasserstein inverse reinforcement learning.
result Wasserstein inverse reinforcement learning enables imitation of expert's reward and solutions in multi-objective optimization.
We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. When the discriminator focuses on task-irrelevant features, it does not provide an informative reward signal, leading to poor task performanc…
NDI learns from expert demonstrations by estimating occupancy measures.
problem Imitation Learning (IL) for complex systems.
method Density estimation of expert's occupancy measure followed by RL.
result NDI achieves state-of-the-art performance on control tasks.
GWIL uses Gromov-Wasserstein distance to align expert and imitation agent states.
problem Cross-domain imitation learning challenges due to different system dimensions and stationary distributions.
method Gromov-Wasserstein Imitation Learning (GWIL) using Gromov-Wasserstein distance.
result GWIL effectively aligns expert and imitation agent states in various continuous control domains.
New method improves imitation learning from expert observations.
problem Challenges in imitation learning from observation setting.
method Reparameterized Variational Divergence Minimization.
result Our method outperforms baseline approaches in low-dimensional tasks.
Imitation from observation is the framework of learning tasks by observing demonstrated state-only trajectories. Recently, adversarial approaches have achieved significant performance improvements over other methods for imitating complex behaviors. However, these adversarial imitation algorithms often require many demo…
New method simplifies and improves imitation learning without adversarial techniques.
problem Stable optimization and convergence issues in adversarial imitation learning methods.
method Proposes a non-adversarial framework for imitation learning, providing stronger convergence guarantees.
result Shows AIRL as a special case and derives new algorithms for offline imitation learning.
Integration of reinforcement learning and imitation learning is an important problem that has been studied for a long time in the field of intelligent robotics. Reinforcement learning optimizes policies to maximize the cumulative reward, whereas imitation learning attempts to extract general knowledge about the traject…
We study risk-sensitive imitation learning where the agent's goal is to perform at least as well as the expert in terms of a risk profile. We first formulate our risk-sensitive imitation learning setting. We consider the generative adversarial approach to imitation learning (GAIL) and derive an optimization problem for…
Proposes a method to imitate active learning heuristics for better performance.
problem The performance of active learning heuristics depends on the classifier model and data structure.
method Imitates the selection of the best active learning heuristic using DAGGER.
result Outperforms state-of-the-art imitation learners and heuristics on well-known datasets.
A new method removes policy optimization in adversarial imitation learning.
problem Adversarial imitation learning's delicate alternated optimization.
method Explicitly condition discriminator on two policies, solving generator's optimization problem directly.
result Simpler approach competitive to prevalent methods.
A new algorithm trains experts to safely guide agents in partially observed environments.
problem Existing imitation learning methods for POMDPs can lead to sub-optimal or unsafe policies.
method Derive an objective to encourage the expert to maximize the agent's reward, then use it to train both expert and agent.
result The algorithm produces an expert policy that the agent can safely imitate, outperforming fixed expert policies.
Recent work has demonstrated that problems-- particularly imitation learning and structured prediction-- where a learner's predictions influence the input-distribution it is tested on can be naturally addressed by an interactive approach and analyzed using no-regret online learning. These approaches to imitation learni…
The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, preferences over demonstrations may be difficult to obtain, and little is known theoretically abo…
Imitation learning trains a policy from expert demonstrations. Imitation learning approaches have been designed from various principles, such as behavioral cloning via supervised learning, apprenticeship learning via inverse reinforcement learning, and GAIL via generative adversarial learning. In this paper, we propose…
Curriculum learning and imitation learning improve control over financial time-series data.
problem Improving control performance over complex financial time-series data.
method Data augmentation for curriculum learning and policy distillation for imitation learning.
result Curriculum learning shows significant improvement over time-series control tasks.
New method uses variational inference to handle confounding in imitation learning.
problem Confounding due to different sensory inputs between expert and imitating agent.
method Train variational inference model to infer expert's latent information and use for latent-conditional policy training.
result Algorithm converges to correct interventional policy and achieves asymptotically optimal performance.
Algorithm improves imitation learning from visual data in partially observable environments.
problem Imitation learning from visual observations with missing expert actions and partial observability.
method Theoretical analysis and Latent Adversarial Imitation from Observations algorithm combining adversarial and latent representations.
result Latent Adversarial Imitation from Observations achieves state-of-the-art performance in high-dimensional robotic tasks.
Dual-structured method improves cross-domain imitation learning.
problem Learning policies in a target domain from source domain expert demonstrations.
method Extracts domain and policy expertness features, synthesizes new features.
result Superior performance compared to other algorithms in MuJoCo tasks.
Recent work on imitation learning has generated policies that reproduce expert behavior from multi-modal data. However, past approaches have focused only on recreating a small number of distinct, expert maneuvers, or have relied on supervised learning techniques that produce unstable policies. This work extends InfoGAI…