arXiv research

The options framework in reinforcement learning models the notion of a skill or a temporally extended sequence of actions. The discovery of a reusable set of skills has typically entailed building options, that navigate to bottleneck states. This work adopts a complementary approach, where we attempt to discover option…

Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor ma…

Paper introduces a new distributional successor measure for reinforcement learning.

Animals need to devise strategies to maximize returns while interacting with their environment based on incoming noisy sensory observations. Task-relevant states, such as the agent's location within an environment or the presence of a predator, are often not directly observable but must be inferred using available sens…

One question central to Reinforcement Learning is how to learn a feature representation that supports algorithm scaling and re-use of learned information from different tasks. Successor Features approach this problem by learning a feature representation that satisfies a temporal constraint. We present an implementation…

A key question in Reinforcement Learning is which representation an agent can learn to efficiently reuse knowledge between different tasks. Recently the Successor Representation was shown to have empirical benefits for transferring knowledge between tasks with shared transition dynamics. This paper presents Model Featu…

This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.

Deep RL approach improves MIS for complex environments.

In this paper, we show how novel transfer reinforcement learning techniques can be applied to the complex task of target driven navigation using the photorealistic AI2THOR simulator. Specifically, we build on the concept of Universal Successor Features with an A3C agent. We introduce the novel architectural contributio…

SF-DQN improves RL transfer by learning successor features.

In this paper we introduce a simple approach for exploration in reinforcement learning (RL) that allows us to develop theoretically justified algorithms in the tabular case but that is also extendable to settings where function approximation is required. Our approach is based on the successor representation (SR), which…

CAST predicts distribution-valued time series by stabilizing and transporting simplex-supported successors.

In classical curve theory, the geometry of a curve in three dimensions is essentially characterized by their invariants, curvature and torsion. When they are given, the problem of finding a corresponding curve is known as 'solving natural equations'. Explicit solutions are known only for a handful of curve classes, inc…

It has been established that diverse behaviors spanning the controllable subspace of an Markov decision process can be trained by rewarding a policy for being distinguishable from other policies \citep{gregor2016variational, eysenbach2018diversity, warde2018unsupervised}. However, one limitation of this formulation is …

Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learning a universal value function (Schaul et al., 2015), which generalizes over goals and states, has previously been shown to be useful for transfer. However, successor features are b…

Proposes method to discover diverse near-optimal policies in reinforcement learning.

A major challenge in reinforcement learning (RL) is the design of agents that are able to generalize across tasks that share common dynamics. A viable solution is meta-reinforcement learning, which identifies common structures among past tasks to be then generalized to new tasks (meta-test). In meta-training, the RL ag…

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free \emph{batch} settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed ex…

In classical curve theory, the geometry of a curve in three dimensions is essentially characterized by their invariants, curvature and torsion. When they are given, the problem of finding a corresponding curve is known as 'solving natural equations'. Explicit solutions are known only for a handful of curve classes, inc…

The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability…

The objective of transfer reinforcement learning is to generalize from a set of previous tasks to unseen new tasks. In this work, we focus on the transfer scenario where the dynamics among tasks are the same, but their goals differ. Although general value function (Sutton et al., 2011) has been shown to be useful for k…

Posterior sampling for reinforcement learning (PSRL) is an effective method for balancing exploration and exploitation in reinforcement learning. Randomised value functions (RVF) can be viewed as a promising approach to scaling PSRL. However, we show that most contemporary algorithms combining RVF with neural network f…

We present a probabilistic model of events in continuous time in which each event triggers a Poisson process of successor events. The ensemble of observed events is thereby modeled as a superposition of Poisson processes. Efficient inference is feasible under this model with an EM algorithm. Moreover, the EM algorithm …

Study homeomorphism groups of ordinals, proving strong distortion and normal generators.

Here we propose using the successor representation (SR) to accelerate learning in a constructive knowledge system based on general value functions (GVFs). In real-world settings like robotics for unstructured and dynamic environments, it is infeasible to model all meaningful aspects of a system and its environment by h…

Identifies optimal base features for zero-shot adaptation in reinforcement learning.

Humans and animals show remarkable flexibility in adjusting their behaviour when their goals, or rewards in the environment change. While such flexibility is a hallmark of intelligent behaviour, these multi-task scenarios remain an important challenge for machine learning algorithms and neurobiological models alike. We…

We develop a normative framework for hierarchical model-based policy optimization based on applying second-order methods in the space of all possible state-action paths. The resulting natural path gradient performs policy updates in a manner which is sensitive to the long-range correlational structure of the induced st…

We seek to align agent policy with human expert behavior in a reinforcement learning (RL) setting, without any prior knowledge about dynamics, reward function, and unsafe states. There is a human expert knowing the rewards and unsafe states based on his preference and objective, but querying that human expert is expens…

Proto-value networks improve deep reinforcement learning representations using auxiliary tasks.

We discuss of the conceptual difficulties connected with the anticommutativity of classical fermion fields, and we argue that the "space" of all classical configurations of a model with such fields should be described as an infinite-dimensional supermanifold M. We discuss the two main approaches to supermanifolds, and …

This paper motivates and develops source traces for temporal difference (TD) learning in the tabular setting. Source traces are like eligibility traces, but model potential histories rather than immediate ones. This allows TD errors to be propagated to potential causal states and leads to faster generalization. Source …

The natural partial ordering of the orbit types of the action of the group of local gauge transformations on the space of connections in space-time dimension d<=4 is investigated. For that purpose, a description of orbit types in terms of cohomology elements of space-time, derived earlier, is used. It is shown that, on…

Develops a bialgebra theory for post-Lie algebras using geometric interpretations and bilinear forms.

Tab-TRM uses recursive model for insurance pricing on tabular data.

Paper describes a state sum formula for a graph coloring polynomial.

Real Milnor fibres become contractible after attaching handles, matching classical results.

Many real-world applications require robust algorithms to learn point processes based on a type of incomplete data --- the so-called short doubly-censored (SDC) event sequences. We study this critical problem of quantitative asynchronous event sequence analysis under the framework of Hawkes processes by leveraging the …

New concept SB-generation helps classify transformation groups.

Composing previously mastered skills to solve novel tasks promises dramatic improvements in the data efficiency of reinforcement learning. Here, we analyze two recent works composing behaviors represented in the form of action-value functions and show that they perform poorly in some situations. As part of this analysi…

Active learning (AL) is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget. In AL one iteratively selects training examples for annotation, often those for which the current model is most uncertain (by some measure). The hope is that active sampling leads to bette…

ARMA cell simplifies neural autoregressive modeling for time series.

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network. However, there are two issues associated with most one-shot NAS methods…

The crowd panic and its contagion play non-negligible roles at the time of the stock crash, especially for China where inexperienced investors dominate the market. However, existing models rarely consider investors in networking stocks and accordingly miss the exact knowledge of how panic contagion leads to abrupt cras…

Empowerment quantifies the influence an agent has on its environment. This is formally achieved by the maximum of the expected KL-divergence between the distribution of the successor state conditioned on a specific action and a distribution where the actions are marginalised out. This is a natural candidate for an intr…

GMC benchmark isolates retrieval in Transformers, revealing max-margin alignment.

In this article, we address the question of how non-knowledge about future events that influence economic agents' decisions in choice settings has been formally represented in economic theory up to date. To position our discussion within the ongoing debate on uncertainty, we provide a brief review of historical develop…

NeuralProphet improves forecast accuracy by 55-92% for short-term forecasts.