arXiv research

New algorithm for reward-free RL with linear function approximation, reducing sample complexity.

We consider stochastic multi-armed bandit problems with complex actions over a set of basic arms, where the decision maker plays a complex action rather than a basic arm in each round. The reward of the complex action is some function of the basic arms' rewards, and the feedback observed may not necessarily be the rewa…

This work characterizes reward function partial identifiability and its impact on policy optimization.

A language for specifying complex reinforcement learning tasks.

Reward-free RL in linear MDPs is as hard as reward-aware RL.

Align-RUDDER improves reinforcement learning with few demonstrations by redistributing rewards.

Enhances reward specification in RL with a novel language-based approach.

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create age…

EPIC quantifies reward differences without policy optimization.

We study the Combinatorial Pure Exploration problem with Continuous and Separable reward functions (CPE-CS) in the stochastic multi-armed bandit setting. In a CPE-CS instance, we are given several stochastic arms with unknown distributions, as well as a collection of possible decisions. Each decision has a reward accor…

New RL algorithm reduces deployment cost for linear function approximations.

Adaptive exploration scheme for evaluating multiple policies with different rewards.

New algorithm for RL with horizon-free reward-free exploration for linear MDPs.

Develops statistical framework for resolving reward function ambiguity in inverse reinforcement learning.

New approach for reward-free exploration reduces estimation error.

To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. …

Neural algorithms optimize arm selection with human preference feedback for complex reward functions.

We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often …

WDAIL uses Wasserstein distance for more effective reward shaping in IL.

Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquir…

AceIRL learns expert reward from active exploration.

Explicit engineering of reward functions for given environments has been a major hindrance to reinforcement learning methods. While Inverse Reinforcement Learning (IRL) is a solution to recover reward functions from demonstrations only, these learned rewards are generally heavily \textit{entangled} with the dynamics of…

This work improves Q-learning for average-reward MDPs, reducing sample and communication complexities in federated settings.

New RL method explores environments without rewards, achieving efficient policy generation.

Fine-tunes diffusion models to generate diverse samples with high genuine rewards.

Bayesian reward learning from demonstrations enables rigorous safety and uncertainty analysis when performing imitation learning. However, Bayesian reward learning methods are typically computationally intractable for complex control problems. We propose Bayesian Reward Extrapolation (Bayesian REX), a highly efficient …

Reinforcement learning in complex environments is a challenging problem. In particular, the success of reinforcement learning algorithms depends on a well-designed reward function. Inverse reinforcement learning (IRL) solves the problem of recovering reward functions from expert demonstrations. In this paper, we solve …

Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be a direct result of goal misspecification, randomness in the reward signal, or correlation of the rew…

Optimizes reward learning design for complex tasks using nonparametric methods.

The paper tackles reward-relevance in offline RL with sparse decision dynamics.

We characterize learnability for stochastic noisy bandits, identifying optimal query complexities.

New RL agent learns sparse rewards efficiently.

New method optimizes policies without assuming known link functions between preferences and rewards.

Inverse reinforcement learning (IRL) is the problem of finding a reward function that generates a given optimal policy for a given Markov Decision Process. This paper looks at an algorithmic-independent geometric analysis of the IRL problem with finite states and actions. A L1-regularized Support Vector Machine formula…

Exploration is widely regarded as one of the most challenging aspects of reinforcement learning (RL), with many naive approaches succumbing to exponential sample complexity. To isolate the challenges of exploration, we propose a new "reward-free RL" framework. In the exploration phase, the agent first collects trajecto…

Reinforcement learning agents are prone to undesired behaviors due to reward mis-specification. Finding a set of reward functions to properly guide agent behaviors is particularly challenging in multi-agent scenarios. Inverse reinforcement learning provides a framework to automatically acquire suitable reward functions…

Optimizes algorithms for non-concave bandit problems.

Task-agnostic RL tackles exploration in MDPs with multiple tasks.

Maximizes Rényi entropy for efficient exploration in reward-free RL.

New method uses explicit human demonstrations to teach missing features in reward learning.

Unified meta algorithms estimate various distribution functionals in infinite-armed bandits.

Reinforcement learning for embodied agents is a challenging problem. The accumulated reward to be optimized is often a very rugged function, and gradient methods are impaired by many local optimizers. We demonstrate, in an experimental setting, that incorporating an intrinsic reward can smoothen the optimization landsc…

Inverse reinforcement learning (IRL) is used to infer the reward function from the actions of an expert running a Markov Decision Process (MDP). A novel approach using variational inference for learning the reward function is proposed in this research. Using this technique, the intractable posterior distribution of the…

Reinforcement learning methods require careful design involving a reward function to obtain the desired action policy for a given task. In the absence of hand-crafted reward functions, prior work on the topic has proposed several methods for reward estimation by using expert state trajectories and action pairs. However…

Safe exploration in RF-RL doesn't increase sample complexity.

This thesis tackles learning reward functions from human comparative feedback.

New algorithm LOOP learns infinite-horizon AMDPs efficiently with function approximation.

Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a partic…