A new approach for exploration in RL using the successor representation.
problem Developing theoretically justified algorithms for exploration in RL.
method The successor representation (SR) and substochastic successor representation (SSR) to incentivize exploration and count observations.
result An algorithm that performs as well as sample-efficient approaches and achieves state-of-the-art performance in Atari games.
Successor Options discovers reusable skills using landmark states.
problem Discovering reusable skills in reinforcement learning.
method Leverages Successor Representations to build a state space model and learns intra-option policies using a novel pseudo-reward.
result Demonstrates the approach's efficacy on grid-worlds and high-dimensional robotic control environments.
A model learns successor representations in uncertain environments.
problem Learning effective strategies in partially observable, noisy environments.
method Neurally plausible model using distributional successor features.
result Distributional successor features support reinforcement learning in noisy environments.
Successor Features improve transfer in RL by decoupling feature and reward.
problem Improving feature representation for task transfer in reinforcement learning.
method Decouples feature representation from reward function, allowing domain transfer.
result Advantages and limitations of Successor Features for transfer identified.
Paper introduces a new distributional successor measure for reinforcement learning.
problem Learning the distributional consequences of behavior in reinforcement learning.
method Formulates distributional successor measure as a distribution over distributions, proposes algorithm to learn it from data.
result Demonstrates zero-shot risk-sensitive policy evaluation.
Deep RL approach improves MIS for complex environments.
problem Improving off-policy evaluation for complex environments.
method Uses successor representation from deep RL to decouple reward and dynamics.
result Empirically stable and applicable to high-dimensional domains.
Model Features improve transfer in reinforcement learning by clustering states.
problem Improving knowledge transfer between tasks with shared transition dynamics.
method Introduces Model Features, a feature representation that clusters behaviourally equivalent states.
result Learning Successor Features is equivalent to learning a Model-Reduction.
Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor ma…
A new method for faster learning in reinforcement learning.
problem Learning from multiple tasks with different goals.
method Universal Successor Representations (USR) and USR Approximator (USRA).
result Agents initialized with USRA trained on USR can achieve goals faster than random initialization.
Algorithm improves transfer learning by inferring successor maps.
problem Machine learning challenges in multi-task scenarios.
method Combining factorized representations and nonparametric memory-based approaches.
result Improves transfer capabilities and outperforms other algorithms.
State2vec improves RL by learning state embeddings that generalize across policies.
problem Inefficient generalization across policies in RL.
method Extends node2vec to learn state embeddings accounting for discounted future state transitions.
result Captures the geometry of the state space, leading to sample-efficient value function approximation.
Develops a new method for optimizing policies in hierarchical models.
problem Optimizing complex policies in hierarchical models.
method Applies second-order methods in the space of state-action paths.
result The natural path gradient method can be computed exactly and reflects state-space hierarchy.
New algorithm learns expert reward structures from batch data.
problem Learning expert reward structures from batch data without dynamics models.
method Deep Successor Feature Networks (DSFN) and transition-regularized imitation network.
result Superior performance on control benchmarks and sepsis management.
SR improves learning speed in dynamic environments.
problem Infeasibility of hand-modeling complex, dynamic systems.
method SR for accelerating learning in GVF-based systems.
result SR improves sample efficiency and learning speed.
This paper develops source traces for faster TD learning.
problem Improving temporal difference learning speed and generalization.
method Introduces source traces as a backward view of successor representations, enabling TD errors to be propagated to potential causal states.
result Demonstrates faster generalization and improved performance of source traces compared to previous methods.
Proto-value networks improve deep reinforcement learning representations using auxiliary tasks.
problem Improving deep reinforcement learning representations with auxiliary tasks.
method Derived a new family of auxiliary tasks based on the successor measure, combined with off-policy learning rule.
result Proto-value networks produce rich features comparable to established algorithms using only linear approximation and a small number of interactions.
VUSFA improves transfer learning for target-driven navigation in AI2THOR.
problem Improving transfer reinforcement learning for complex visual navigation tasks.
method Introducing SFDP and Variational Information Bottlenecks to A3C agent.
result VUSFA achieves state-of-the-art performance and generalizability.
VISR learns controllable features for fast task inference.
problem Generalizing behaviors beyond explicitly learned set for subsequent tasks.
method Combines Successor Features and Variational Inference.
result Achieves human-level performance on 14 Atari games.
SF-DQN improves RL transfer by learning successor features.
problem Transfer RL with shared dynamics but different reward functions.
method Decomposes Q-function into SF and reward mapping; uses GPI for policy improvement.
result SF-DQN with GPI converges faster and generalizes better than traditional RL methods.
CAST predicts distribution-valued time series by stabilizing and transporting simplex-supported successors.
problem Forecasting distribution-valued time series with structural failure modes.
method CAST (Causal Anchored Simplex Transport) uses successors retrieved from causal context, stabilized with a persistence anchor, and locally transported on ordered supports.
result CAST outperforms baselines on eleven public and simulated benchmarks, achieving best average rank on both one-step KL and autoregressive rollout JSD.
In classical curve theory, the geometry of a curve in three dimensions is essentially characterized by their invariants, curvature and torsion. When they are given, the problem of finding a corresponding curve is known as 'solving natural equations'. Explicit solutions are known only for a handful of curve classes, inc…
USFs capture dynamics for faster RL task transfer.
problem Applying knowledge from one task to another.
method Proposed Universal Successor Features (USFs) for RL.
result USFs accelerate training and transfer knowledge.
New framework for efficient query-based imitation learning.
problem Aligning agent policy with human expert behavior without prior knowledge.
method Adversarial reward query with successor representation.
result Significantly outperforms uncertainty-based methods in query efficiency.
This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.
problem Generalization from easy to hard tasks, especially out-of-domain generalization.
method Formalizes inductive generalization, introduces Inductive Learning, and outlines steps to adapt techniques for learning model successors.
result A new learning paradigm (Inductive Learning) that emphasizes induction and universal properties of learning and computation.
Proposes method to discover diverse near-optimal policies in reinforcement learning.
problem Finding different solutions to the same problem in reinforcement learning.
method Formalizes problem as CMDP, uses Successor Features, proposes new diversity rewards.
result Proposed method discovers diverse near-optimal policies that are robust and distinct.
USFAs combine UVFAs, SFs, and GPI for scalable, instant RL generalisation.
problem Generalizing to unseen tasks in reinforcement learning.
method Combining universal value function approximators, successor features, and generalized policy improvement.
result Demonstrates practical benefits and transfer abilities in a complex 3D environment.
In classical curve theory, the geometry of a curve in three dimensions is essentially characterized by their invariants, curvature and torsion. When they are given, the problem of finding a corresponding curve is known as 'solving natural equations'. Explicit solutions are known only for a handful of curve classes, inc…
This work improves understanding of reinforcement learning state representations.
problem Lack of precise characterization of how and when state representations generalize.
method Developed a bound on the generalization error based on effective dimension.
result Bound quantifies the tension between generalization and approximation.
Develops a bialgebra theory for post-Lie algebras using geometric interpretations and bilinear forms.
problem Characterizing and understanding post-Lie algebras and their associated structures.
method Utilizes Manin triples and generalized Hessian Lie groups to define and characterize post-Lie algebras with nondegenerate symmetric invariant bilinear forms.
result Establishes a bialgebra theory for post-Lie algebras via the Manin triple approach, including new algebraic structures like pp-post-Lie algebras.
Our work proves CSF can recover ground-truth features in RL, improving understanding of feature learning.
problem Understanding the role of representation and mutual information in reinforcement learning.
method Investigates Contrastive Successor Features (CSF) method for identifiable representation learning in reinforcement learning.
result Proves CSF can recover ground-truth features up to a linear transformation.
New method improves data efficiency in reinforcement learning by composing skills.
problem Improving data efficiency in reinforcement learning by composing previously mastered skills.
method Extending policy improvement to maximum entropy framework, introducing successor features, and explicitly learning divergence between base policies.
result Proposes a novel approach that outperforms or matches existing methods in various tasks.
Study homeomorphism groups of ordinals, proving strong distortion and normal generators.
problem Understanding algebraic and geometric properties of homeomorphism groups of ordinals.
method Analyzing successor ordinals with connections to permutation groups and manifolds.
result Proves strong distortion and normal generators for homeomorphism groups of ordinals.
SU improves exploration in reinforcement learning, surpassing human performance on Atari games.
problem Challenges in scaling PSRL for reinforcement learning with neural networks.
method Design and implementation of Successor Uncertainties (SU) algorithm.
result SU outperforms human performance on Atari games and surpasses RVF competitor Bootstrapped DQN.
The paper shows how neural networks can approximate PDEs with polynomial scaling in dimension.
problem Understanding the complexity of approximating PDE solutions with neural networks.
method Developed a proof technique to simulate gradient descent using neural networks.
result Neural network parameters scale polynomially with input dimension for approximating PDE solutions.
Identifies optimal base features for zero-shot adaptation in reinforcement learning.
problem Unclear what constitutes a good set of base features for a wide range of downstream tasks.
method Identifies optimal base features based on downstream performance, without assuming downstream tasks are linear.
result Optimal base features are the same across three task families, differing from Laplacian eigenfunctions.
Paper proposes a new method for efficient exploration in reinforcement learning.
problem Sparse reward reinforcement learning challenges in exploration.
method Learn separate intrinsic and extrinsic task policies, schedule between them, and use successor feature control (SFC).
result Substantially improved exploration efficiency with SFC and hierarchical usage of intrinsic drives.
We present a probabilistic model of events in continuous time in which each event triggers a Poisson process of successor events. The ensemble of observed events is thereby modeled as a superposition of Poisson processes. Efficient inference is feasible under this model with an EM algorithm. Moreover, the EM algorithm …
The natural partial ordering of the orbit types of the action of the group of local gauge transformations on the space of connections in space-time dimension d<=4 is investigated. For that purpose, a description of orbit types in terms of cohomology elements of space-time, derived earlier, is used. It is shown that, on…
This paper provides statistical guarantees for WAE's latent space regeneration.
problem Lack of statistical analysis for Autoencoders, especially WAE.
method Utilizes Vapnik Chervonenkis (VC) theory and Optimal Transport of measures under the Wasserstein metric.
result WAE achieves the target distribution in the latent space and regenerates the input distribution.
Paper describes a state sum formula for a graph coloring polynomial.
problem Counting n-face colorings of ribbon graphs for various n. method Combines topological quantum field theory and diagrammatic tensors.
result Describes a state sum formula for the total face color polynomial.
The paper proposes a method to transfer skills between tasks using disentangled latent policies.
problem Transfer learning in reinforcement learning struggles with diverse tasks without explicit supervision.
method Learning a small set of policies in a disentangled latent space that can be recombined to solve many tasks.
result Disentangled latent policies enable quick performance on many diverse tasks.
We discuss of the conceptual difficulties connected with the anticommutativity of classical fermion fields, and we argue that the "space" of all classical configurations of a model with such fields should be described as an infinite-dimensional supermanifold M. We discuss the two main approaches to supermanifolds, and …
Real Milnor fibres become contractible after attaching handles, matching classical results.
problem Topology of real isolated hypersurface singularities.
method Proving real Milnor fibres contractible after attaching handles, defining real vanishing cycles.
result Integer homology groups of real Milnor fibres are isomorphic to those of a bouquet of spheres under certain conditions.
New concept SB-generation helps classify transformation groups.
problem Classifying transformation groups through quasi-isometry invariants.
method Identifying SB-generated groups in specific transformation groups.
result SB-generation provides robust extension of finite generation.
Method synthesizes incomplete event sequences to learn Hawkes processes.
problem Learning Hawkes processes from short doubly-censored event sequences.
method Sampling-stitching data synthesis method for SDC event sequences.
result Improves learning results for both time-invariant and time-varying Hawkes processes.
Empowerment quantifies the influence an agent has on its environment. This is formally achieved by the maximum of the expected KL-divergence between the distribution of the successor state conditioned on a specific action and a distribution where the actions are marginalised out. This is a natural candidate for an intr…
Tab-TRM uses recursive model for insurance pricing on tabular data.
problem Insurance pricing on tabular data.
method Adapts recursive latent reasoning to insurance modeling using a compact, parameter-efficient network.
result Improves insurance pricing accuracy using iterative refinement of latent tokens.
In this article, we address the question of how non-knowledge about future events that influence economic agents' decisions in choice settings has been formally represented in economic theory up to date. To position our discussion within the ongoing debate on uncertainty, we provide a brief review of historical develop…