Improves reinforcement learning extrapolation in Gridworlds.
problem Generalizing to unseen states in reinforcement learning.
method Avoiding deterministic action choice, using ego-centric representation, incorporating symmetry, and adding an entropy term.
result Significant improvement in extrapolation performance in Gridworld environments.
New method detects dangerous states in gridworlds using geometric defects.
problem Detecting dangerous states in gridworlds for AI safety.
method Modified state complexes and Gromov's Link Condition.
result Geometric defects indicate dangerous states in gridworlds.
Designs a framework to transfer causal models between similar environments.
problem Transferability of causal models between different but similar environments.
method Object-oriented representations and continuous optimization for structure learning.
result Demonstrates advantages in gridworld settings using reinforcement learning.
Algorithm learns actions from past states in complex tasks.
problem Learning policies from human feedback is expensive.
method Combining learned feature encoder with inverse models to simulate past actions.
result Algorithm can infer specific skills from single state.
New methods for evaluating and optimizing policies in offline RL with unobserved confounders.
problem Evaluating and optimizing policies in the presence of unobserved confounders.
method Characterized settings and algorithms for consistent value estimates and lower bounds, with sample complexity guarantees.
result Proved local convergence guarantees for offline policy improvement.
Direct and indirect RL methods classified and compared.
problem Classifying RL methods for sequential decision making.
method Direct RL solves optimal policy directly, indirect RL solves Bellman equation.
result Direct and indirect RL methods are equivalent and can be unified.
Exploration and adaptation to new tasks in a transfer learning setup is a central challenge in reinforcement learning. In this work, we build on the idea of modeling a distribution over policies in a Bayesian deep reinforcement learning setup to propose a transfer strategy. Recent works have shown to induce diversity i…
One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity be…
Autonomous agents trained via reinforcement learning present numerous safety concerns: reward hacking, negative side effects, and unsafe exploration, among others. In the context of near-future autonomous agents, operating in environments where humans understand the existing dangers, human involvement in the learning p…
Current reinforcement learning methods fail if the reward function is imperfect, i.e. if the agent observes reward different from what it actually receives. We study this problem within the formalism of Corrupt Reward Markov Decision Processes (CRMDPs). We show that if the reward corruption in a CRMDP is sufficiently "…
Multi-task Inverse Reinforcement Learning (IRL) is the problem of inferring multiple reward functions from expert demonstrations. Prior work, built on Bayesian IRL, is unable to scale to complex environments due to computational constraints. This paper contributes a formulation of multi-task IRL in the more computation…
Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the non-expansion property, and its direct use may fail to converge even in value iteration. In this pape…
Agents learn to communicate and solve navigation tasks efficiently.
problem Understanding and developing communication in artificial agents.
method Multi-agent navigation tasks in gridworld environments.
result Agents develop an interpretable communication protocol that optimally solves tasks.
Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. U…
PCHID improves sample efficiency in reinforcement learning tasks.
problem Sparse rewards make learning policies difficult in reinforcement learning.
method Hindsight Inverse Dynamics with Hindsight Experience Replay and Policy Continuation.
result PCHID significantly improves sample efficiency and final performance on multi-goal tasks.
In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative f…
Active IRL selects optimal human demonstrations for learning AI preferences.
problem Costly human demonstrations in IRL for autonomous systems.
method Information-theoretic acquisition function for selecting informative human demonstrations.
result Efficiently reduces human effort in learning AI preferences.
UTE improves reinforcement learning by measuring action uncertainty, enhancing policy learning efficiency.
problem Degrading performance of action repetition in reinforcement learning, especially with sub-optimal actions.
method UTE uses ensemble methods to measure uncertainty during action extension, allowing strategic exploration or certainty.
result UTE outperforms existing action repetition algorithms, significantly enhancing policy learning efficiency.
Before deploying autonomous agents in the real world, we need to be confident they will perform safely in novel situations. Ideally, we would expose agents to a very wide range of situations during training, allowing them to learn about every possible danger, but this is often impractical. This paper investigates safet…
The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence. In this paper, we consider the problem of learning to simultaneously transfer across both environments (ENV) and tasks (TASK), probably more importantly, by learning from only sparse (ENV, TASK) pairs …
Efficient exploration is necessary to achieve good sample efficiency for reinforcement learning in general. From small, tabular settings such as gridworlds to large, continuous and sparse reward settings such as robotic object manipulation tasks, exploration through adding an uncertainty bonus to the reward function ha…
How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source…
Algorithm learns mixtures of Markov chains and MDPs from short trajectories.
problem Learning mixtures of Markov chains and MDPs from short unlabeled trajectories.
method Subspace estimation, spectral clustering, EM algorithm, model estimation, classification.
result 96.6% average accuracy on a mixture of two MDPs in gridworld, outperforming EM algorithm with random initialization.
Goals for reinforcement learning problems are typically defined through hand-specified rewards. To design such problems, developers of learning algorithms must inherently be aware of what the task goals are, yet we often require agents to discover them on their own without any supervision beyond these sparse rewards. W…
A new method for learning policies in multiple environments.
problem Learning policies that work in different but related environments.
method Decentralized entropy-regularized policy gradient method.
result The method can learn effective policies in multiple environments.
Eigenoptions improve credit assignment in reinforcement learning.
problem Improving credit assignment in reinforcement learning models.
method Investigated eigenoptions for credit assignment in model-free RL, comparing pre-specified and online discovery methods.
result Pre-specified eigenoptions aid exploration and credit assignment, while online discovery can hinder learning.
New algorithm infers reward function from agent's learning trajectories.
problem Inferring reward function from agent's learning data.
method Gradient-based approach to recover reward function.
result Improved performance compared to state-of-the-art methods.
This work improves policy optimization by maximizing entropy of state distribution, leading to better exploration.
problem Lack of exploration in state space when maximizing policy entropy.
method Proposes maximizing the entropy of a lower bound approximation to the state weighting distribution, based on latent space representation.
result Entropy regularization based on marginal state distribution achieves superior state space coverage and better performance in various domains.
CoCoRL learns safe constraints from demonstrations with unknown rewards.
problem Learning safe constraints from demonstrations with different unknown rewards.
method Convex Constraint Learning for Reinforcement Learning (CoCoRL) constructs a convex safe set based on demonstrations.
result CoCoRL learns constraints that lead to safe driving behavior and can safely transfer to different tasks and environments.
NOHD optimizes multi-agent systems by decomposing dynamics into irrotational and solenoidal components.
problem Non-stationarity and conflicting interests in multi-agent learning problems.
method NOHD (Newton Optimization on Helmholtz Decomposition) decomposes system dynamics into irrotational and solenoidal components.
result NOHD ensures quadratic convergence in purely irrotational and solenoidal systems and attracts to stable fixed points in general multi-agent systems.
USFs capture dynamics for faster RL task transfer.
problem Applying knowledge from one task to another.
method Proposed Universal Successor Features (USFs) for RL.
result USFs accelerate training and transfer knowledge.
Combines BC and GAIL for efficient imitation learning.
problem Efficient imitation learning without reward signals.
method Integrates Behavior Cloning and Generative Adversarial Imitation Learning.
result Combination leads to stable and sample-efficient learning.
ISA learns subgoals for reinforcement learning agents.
problem Learning subgoals for efficient reinforcement learning.
method Induces a subgoal automaton from observation traces using inductive logic programming.
result ISA learns subgoals that improve RL performance and convergence.
Although reinforcement learning (RL) can provide reliable solutions in many settings, practitioners are often wary of the discrepancies between the RL solution and their status quo procedures. Therefore, they may be reluctant to adapt to the novel way of executing tasks proposed by RL. On the other hand, many real-worl…
ADVISOR dynamically balances imitation and reinforcement learning to overcome the imitation gap.
problem The gap between imitation learning and reinforcement learning when teaching agents have privileged information.
method Adaptive Insubordination (ADVISOR) dynamically weights imitation and reward-based reinforcement learning losses.
result On-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, and their combinations.
There is a long history of using meta learning as representation learning, specifically for determining the relevance of inputs. In this paper, we examine an instance of meta-learning in which feature relevance is learned by adapting step size parameters of stochastic gradient descent---building on a variety of prior w…
This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Boo…
OtoWorld helps agents learn to navigate by listening in interactive environments.
problem Training agents to navigate using auditory information.
method Interactive environment with agents learning to listen and navigate.
result Agents can win at OtoWorld, a navigation game with sound sources.
In this work we present a novel approach for transfer-guided exploration in reinforcement learning that is inspired by the human tendency to leverage experiences from similar encounters in the past while navigating a new task. Given an optimal policy in a related task-environment, we show that its bisimulation distance…
This paper introduces sample-averaged Q-learning for better RL performance.
problem Improving reinforcement learning algorithms by managing uncertainty.
method Integrates statistical inference into Q-learning through sample averaging and functional central limit theorem.
result Establishes a unified theoretical foundation for sample-averaged Q-learning.
New method learns high-quality Laplacian representations for reinforcement learning.
problem Lack of accurate Laplacian representations in large or continuous state spaces.
method Reformulated spectral graph drawing objective to have eigenvectors as unique global minimizer.
result Learned Laplacian representations more faithfully approximate the ground truth.
Paper proposes a novel RRL framework that learns from images and incorporates expert knowledge.
problem Lack of effective methods to incorporate expert background knowledge and learn from non-relational data in RRL.
method Differentiable Inductive Logic Programming (ILP) for learning relational information from images and incorporating expert knowledge.
result Efficacy demonstrated on various environments and datasets, showing improved learning and generalization.
The paper introduces a method to learn Markov state abstractions for reinforcement learning.
problem Learning Markov state representations in complex environments.
method The paper introduces a novel set of conditions and a training procedure combining inverse model estimation and temporal contrastive learning.
result The approach learns representations that capture the underlying structure of the domain and improve sample efficiency.
This project proposes using reinforcement learning to train spiking neural networks.
problem Training spiking neural networks using traditional methods is challenging due to the discrete nature of spikes.
method The project investigates two approaches: 1) treating each neuron as an RL agent, 2) applying the reparameterization trick.
result The project demonstrates that reinforcement learning can be applied to train spiking neural networks.
Proves error bounds for state representation in RL using graph spectral features.
problem Addressing the curse of dimensionality in RL with unknown transition graphs.
method Proves upper bounds on approximation error of linear value function approximation using learned spectral features of the state-graph.
result Error bounds scale with algebraic connectivity and eigenvector estimation error.
Platform combines RL and language models to study narrative influence on AI decisions.
problem Understanding how narrative elements shape AI decision-making.
method Dual-system architecture with reinforcement learning and language model integration.
result Initial experiments show narrative frameworks can influence AI decision-making.
A new method for student-initiated action advice using novelty detection.
problem Exploration and sample inefficiency in RL, especially with teacher absence.
method Random Network Distillation (RND) to measure advice novelty, updates only for advised states.
result Significant performance improvement over state-of-the-art methods, especially in challenging scenarios.
This paper interpolates reward functions to predict optimal value functions in MORL.
problem Finding optimal value functions in MORL requires recomputing for each set of weights.
method Interpolating reward function weights to smooth value function transformations.
result Smooth interpolation of optimal value functions over reward function weights.