VILD learns policies from mixed-quality demonstrations by modeling expert levels.
problem Challenges in learning from diverse-quality demonstrations.
method Explicitly models expert levels with a probabilistic graphical model and variational approach.
result VILD outperforms state-of-the-art methods in continuous-control benchmarks.
Develops deep learning for optimizing 5G radio resource allocation.
problem Optimizing 5G base station radio resources for diverse QoS requirements.
method Cascaded neural network structure with deep transfer learning for non-stationary conditions.
result Cascaded neural networks outperform fully connected neural networks in QoS guarantee.
Efficient neural network ensembles improve image classification reliability and uncertainty quantification.
problem Uncertainty in neural network predictions for industrial image classification.
method Investigated efficient neural network ensembles (snapshot, batch, multi-input multi-output) for image classification reliability and uncertainty quantification.
result Batch ensemble is a cost-effective and competitive alternative to deep ensembles, offering savings in training and test time.
Unified framework for portfolio optimization using multiple hypotheses.
problem Risk diversification in portfolio allocation.
method Structured ensemble learning approach with diversity control.
result Structured ensembles link predictor diversity to risk diversification.
The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, preferences over demonstrations may be difficult to obtain, and little is known theoretically abo…
LEARN-SAM improves RL from sub-optimal demonstrations by localizing expert policies and selectively using demonstrations.
problem Improving RL from sub-optimal or sparse demonstrations.
method Local Ensemble and Reparameterization with Split and Merge of expert policies (LEARN-SAM).
result LEARN-SAM boosts learning speed and accuracy by selectively using demonstrations.
Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, we propose a novel approach that utilizes confidence scores, which describe the quality of demonstrat…
A framework helps reinforcement learning agents understand and decompose tasks from human demonstrations.
problem Sparse-reward tasks where demonstrations are used as sources of causal knowledge.
method Develops causal models through observation and reasons from this knowledge to decompose tasks.
result A basic implementation of Reasoning from Demonstration (RfD) is effective in sparse-reward tasks.
Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from …
Novel framework learns policies from noisy expert demonstrations.
problem Learning effective policies with noisy expert demonstrations.
method Adaptive learning framework that jointly interacts with the environment and expert demonstrations, assigning weights to filter out noisy demonstrations.
result The proposed approach learns robustly with noisy demonstrations and achieves higher performance in fewer iterations.
Recent research has shown that although Reinforcement Learning (RL) can benefit from expert demonstration, it usually takes considerable efforts to obtain enough demonstration. The efforts prevent training decent RL agents with expert demonstration in practice. In this work, we propose Active Reinforcement Learning wit…
Enhances demonstrations with safety specifications using LTL.
problem Learning from demonstrations without safety specifications.
method Converts LTL specifications into a differentiable loss for training.
result System learns dynamic movement primitives that satisfy complex specifications.
Method learns reusable actions from human demonstrations.
problem Learning reusable actions from human demonstrations.
method Skill segmentation, inverse reinforcement learning, one-class support vector machine.
result Inferred options improve learning and planning.
SAIL learns from sub-optimal demonstrations to improve sample efficiency in sparse reward tasks.
problem Reducing sample complexity in sparse-rewarded tasks.
method Self-Adaptive Imitation Learning (SAIL) that exploits sub-optimal demonstrations and efficient exploration.
result Significantly improved sample efficiency and better final performance across various tasks.
SPReD uses uncertainty to decide imitation from demonstrations.
problem Learning from sparse rewards with few demonstrations.
method Ensemble methods to model Q-value distributions, probabilistic and advantage-based uncertainty quantification.
result Significant gains in reinforcement learning across multiple tasks.
A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may ha…
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging e…
CoCoRL learns safe constraints from demonstrations with unknown rewards.
problem Learning safe constraints from demonstrations with different unknown rewards.
method Convex Constraint Learning for Reinforcement Learning (CoCoRL) constructs a convex safe set based on demonstrations.
result CoCoRL learns constraints that lead to safe driving behavior and can safely transfer to different tasks and environments.
Bayesian Experience Reuse improves learning from multiple experts.
problem Learning from multiple experts with conflicting goals.
method Bayesian neural networks with shared features to model uncertainty and derive a probability distribution over expert models.
result BERS method effectively samples demonstrations from the derived distribution to reuse them in new tasks.
Paper proposes a method to learn and exceed expert demonstrations in unknown reward environments.
problem Learning to outperform expert demonstrations in unknown reward environments.
method A novel concurrent reward and action policy learning approach with a stereo utility definition.
result The proposed method can outperform expert demonstrations in various environments.
Estimates rewards from a demonstrator's learning process.
problem Estimating rewards from a demonstrator's behavior.
method Leveraging the demonstrator's exploration phase for reward estimation.
result Consistent reward estimation possible without identifiability issues.
Paper improves RL from imperfect demonstrations with soft expert guidance.
problem Improper and insufficient expert demonstrations in RLfD.
method Formalizes imperfect expert setting, tackles optimality and convergence issues with soft constraints, and uses local linear search on dual form.
result Method achieves consistent improvement over other RLfD methods.
Method learns from video demonstrations with human feedback.
problem Teaching autonomous agents using video demonstrations and human feedback.
method Constructs a mapping between standard and visual representations using a neural network.
result Effective in teaching a hopper agent to perform a backflip with minimal human feedback.
A model learns rewards from diverse demonstrations for structurally similar tasks.
problem Difficulty in learning reward functions from demonstrations in real-world applications.
method Deep latent variable model that infers rewards from heterogeneous demonstrations of related tasks.
result Model can infer rewards for new tasks from a single demonstration.
New method learns to answer questions from correct demonstrations without assuming bounded complexity of the demonstrator.
problem Learning to generate answers from correct demonstrations with multiple correct answers.
method Formalizes imitation learning in contextual bandits, relying on reward model complexity, not policy complexity.
result Achieves nearly optimal performance with sample complexity logarithmic in reward class cardinality.
New algorithm robustly learns from corrupted demonstrations, even with constant fraction of noise.
problem Learning from corrupted demonstrations where a fraction of data is noise or outliers.
method Proposes a novel robust algorithm using a Median-of-Means (MOM) objective.
result Guarantees accurate policy estimation even with constant fraction of outliers, similar to classical methods in expert demonstration settings.
Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decision-making task. We fo…
Imitation learning targets deriving a mapping from states to actions, a.k.a. policy, from expert demonstrations. Existing methods for imitation learning typically require any actions in the demonstrations to be fully available, which is hard to ensure in real applications. Though algorithms for learning with unobservab…
Method distills reward and strategies from diverse demonstrators.
problem Reward ambiguity and heterogeneity in human demonstrations.
method Reward network distillation to infer task goal and strategies.
result Better recovery of task and strategy rewards.
Identifies latent actions and dynamics from offline data with diverse demonstrators.
problem Recovering latent actions and environment dynamics from action-free trajectories.
method Assumes distinct policies for each demonstrator, identifies latent transitions and policies via matrix factorization.
result Identifies latent transitions and demonstrator policies up to permutation.
Align-RUDDER improves reinforcement learning with few demonstrations by redistributing rewards.
problem Learning complex tasks with sparse and delayed rewards using few demonstrations.
method Align-RUDDER uses a profile model for reward redistribution based on multiple sequence alignment of demonstrations.
result Align-RUDDER outperforms competitors on complex tasks with few demonstrations.
Improves online learning with expert demonstrations, quality matters.
problem Improving online learning through offline demonstration data.
method Thompson sampling applied to a multi-armed bandit model, informed by expert demonstrations and Bayes' rule.
result Substantial empirical regret reduction with expert demonstrations, improving online performance.
Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…
PODNet discovers plannable options from unstructured demonstrations.
problem Learning from unstructured, multi-objective demonstrations.
method Custom categorical variational autoencoder, recurrent option inference network, option-conditioned policy network, and option dynamics model.
result PODNet enables learning from demonstration for multiple tasks and planning.
Teaches manifolds from teacher's structured data.
problem Learning a manifold from teacher's structured data.
method Extends existing approaches to learning from randomly sampled data points, considering structured data provided by a teacher.
result Demonstrations can significantly reduce data points needed for manifold learning.
Imitation from observation (IfO) is the problem of learning directly from state-only demonstrations without having access to the demonstrator's actions. The lack of action information both distinguishes IfO from most of the literature in imitation learning, and also sets it apart as a method that may enable agents to l…
The paper learns robot skills from demonstrations without supervision.
problem Discovering robotic options from unlabelled demonstrations.
method Temporal variational inference for latent variable learning.
result The framework can learn options across multiple datasets.
Algorithm identifies intended fairness constraints from expert demonstrations for fair clustering.
problem Fair clustering challenges due to incomplete fairness constraints.
method Algorithm identifies fairness metric from expert demonstrations and generates clusters.
result Algorithm identifies and generates fair clusters from limited expert demonstrations.
Demonstration-regularized RL reduces sample complexity for policy identification.
problem Improving reinforcement learning's sample efficiency with expert demonstrations.
method KL-regularization of behavior cloning using expert demonstrations.
result Demonstration-regularized RL achieves optimal policy identification with reduced sample complexity.
New RL method handles diverse human operator data.
problem Handling data from multiple, unknown policies.
method Domain-Invariant Model-based Offline RL (DIMORL) with Risk Extrapolation (REx).
result Models trained with REx generalize better across different demonstrators.
Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated action…
Pretraining reinforcement learning methods with demonstrations has been an important concept in the study of reinforcement learning since a large amount of computing power is spent on online simulations with existing reinforcement learning algorithms. Pretraining reinforcement learning remains a significant challenge i…
Compared to reinforcement learning, imitation learning (IL) is a powerful paradigm for training agents to learn control policies efficiently from expert demonstrations. However, in most cases, obtaining demonstration data is costly and laborious, which poses a significant challenge in some scenarios. A promising altern…
Paper proposes a curriculum for RL agents to learn from expert demonstrations.
problem Training RL agents efficiently from expert demonstrations.
method Develops a training curriculum that starts from the goal and works backwards.
result Improves sample efficiency for some tasks compared to PPO.
We propose Generative Predecessor Models for Imitation Learning (GPRIL), a novel imitation learning algorithm that matches the state-action distribution to the distribution observed in expert demonstrations, using generative models to reason probabilistically about alternative histories of demonstrated states. We show …
A new reward learning module improves imitation learning in high-dimensional environments.
problem Challenges in high-dimensional environments for imitation learning.
method Generative model to generate intrinsic reward signals.
result Our method outperforms state-of-the-art IRL methods on Atari games.
A novel RL approach learns robotic manipulation without human demonstrations.
problem Learning robotic manipulation policies efficiently and effectively.
method Introducing simulated locomotion demonstration rewards (SLDRs) to enable RL learning.
result The approach achieves higher success rates and faster learning compared to alternatives.
Paper tackles robust imitation learning from noisy demonstrations.
problem Learning from noisy demonstrations is challenging.
method Optimizes a classification risk with a symmetric loss, combining pseudo-labeling and co-training.
result Our method is more robust than state-of-the-art methods.