Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is n…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Hierarchical Policies”
Develops a new method for optimizing policies in hierarchical models.
problem Optimizing complex policies in hierarchical models.
method Applies second-order methods in the space of state-action paths.
result The natural path gradient method can be computed exactly and reflects state-space hierarchy.
Proposes a model for multi-agent reinforcement learning with hierarchical graph attention network.
problem Limited transferability of trained policies to new multi-agent tasks.
method Uses hierarchical graph attention network for representation learning and multi-agent actor-critic for policy learning.
result Demonstrates superior performance in mixed cooperative and competitive tasks compared to existing methods.
Learning an optimal policy from a multi-modal reward function is a challenging problem in reinforcement learning (RL). Hierarchical RL (HRL) tackles this problem by learning a hierarchical policy, where multiple option policies are in charge of different strategies corresponding to modes of a reward function and a gati…
In this work, we provide theoretical guarantees for reward decomposition in deterministic MDPs. Reward decomposition is a special case of Hierarchical Reinforcement Learning, that allows one to learn many policies in parallel and combine them into a composite solution. Our approach builds on mapping this problem into a…
Variational inference improves hierarchical imitation learning of control programs.
problem Learning structured control policies from demonstrations.
method Variational inference for discovering hierarchical structure in observation-action traces.
result Variational inference leads to more efficient and generalized control policies.
We present Multitask Soft Option Learning(MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This ''soft'' version of options avoids several instabilities during training in a …
The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. …
Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can le…
Hierarchical decoupling improves sample efficiency for complex robots.
problem Learning long-range behaviors on complex robots.
method Two-part policy: low-level imitation and high-level transfer, with KL regularization.
result Hierarchical transfer significantly improves zero-shot high-level transfer and stabilizes learning.
In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies. We propose an unsupervised learning scheme, based on asymmetric self-play from Sukhbaatar et al. (2018), that automatically learns a good representation of sub-goals in the environment and a low-level policy that can…
Reinforcement learning agents that operate in diverse and complex environments can benefit from the structured decomposition of their behavior. Often, this is addressed in the context of hierarchical reinforcement learning, where the aim is to decompose a policy into lower-level primitives or options, and a higher-leve…
Proves EM algorithm guarantees for hierarchical imitation learning.
problem Learning hierarchical policies from expert demonstrations.
method Converts hierarchical imitation learning to latent variable model inference, analyzes EM algorithm.
result Proves EM algorithm converges to true parameter under regular conditions.
A new benchmark task for evaluating policy learning in complex, high-dimensional action spaces.
problem Lack of a commonly accepted benchmark for evaluating policy learning in hierarchical tasks with high-dimensional action spaces.
method Proposed DinerDash Gym benchmark and Decomposed Policy Graph Modelling (DPGM) algorithm.
result DPGM achieves significant improvement over baselines and effectively injects domain knowledge.
MGHRL learns to generate high-level meta strategies for new tasks.
problem Efficiency and generalization in meta-RL for wide task distributions.
method Generates high-level meta strategies over subgoals, leaving subtask learning independent.
result More efficient and generalized meta-learning from past experience.
In this work, we take a representation learning perspective on hierarchical reinforcement learning, where the problem of learning lower layers in a hierarchy is transformed into the problem of learning trajectory-level generative models. We show that we can learn continuous latent representations of trajectories, which…
Option discovery and skill acquisition frameworks are integral to the functioning of a Hierarchically organized Reinforcement learning agent. However, such techniques often yield a large number of options or skills, which can potentially be represented succinctly by filtering out any redundant information. Such a reduc…
Unified pair trading approach using hierarchical reinforcement learning.
problem Decoupling pair selection and trading leads to limited performance.
method Hierarchical reinforcement learning framework for joint pair selection and trading.
result Unified approach outperforms existing methods on real-world stock data.
MAVEN improves multi-agent exploration by hybridizing value and policy-based methods.
problem Exploration and suboptimality in complex multi-agent environments.
method MAVEN combines value and policy-based approaches with a latent space for hierarchical control.
result MAVEN achieves significant performance improvements on challenging multi-agent tasks.
H-ReIL learns to drive safely in near-accident scenarios.
problem Driving safely in high-risk near-accident situations.
method Hierarchical RL and IL approach.
result High-level policy switches between low-level policies for safe driving.
We introduce Compositional Imitation Learning and Execution (CompILE): a framework for learning reusable, variable-length segments of hierarchically-structured behavior from demonstration data. CompILE uses a novel unsupervised, fully-differentiable sequence segmentation module to learn latent encodings of sequential d…
Improved cooperation between levels boosts reinforcement learning performance.
problem Training multi-level policies in hierarchical reinforcement learning.
method Modeling policy optimization as a multi-agent process and inducing cooperation between sub-policies.
result Inducing cooperation between sub-policies leads to stronger and more sample-efficient policies.
Method solves long-horizon robotic tasks via imitation and reinforcement learning.
problem Long-horizon robotic tasks with complex sequences of actions.
method Two-phase approach: imitation learning followed by reinforcement learning.
result Method can scale to challenging long-horizon tasks.
Policy-gradient method controls multiple non-cohesive targets.
problem Controlling multiple non-cohesive targets in a decentralized manner.
method Proximal Policy Optimization for target selection and driving.
result Effective control of non-cohesive targets without prior dynamics knowledge.
Building agents that can explore their environments intelligently is a challenging open problem. In this paper, we make a step towards understanding how a hierarchical design of the agent's policy can affect its exploration capabilities. First, we design EscapeRoom environments, where the agent must figure out how to n…
The paper introduces an adjacency constraint to improve goal-conditioned HRL.
problem Training inefficiency in goal-conditioned HRL due to large action space.
method Restricting the high-level action space to a k-step adjacent region of the current state.
result The adjacency constraint preserves optimal hierarchical policies and improves HRL performance.
Director learns hierarchical behaviors from pixels, outperforming exploration methods.
problem Challenges in learning long-term hierarchical behaviors from pixels.
method Directly learns hierarchical behaviors from pixels using a latent space world model.
result Director outperforms exploration methods on various tasks.
We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acq…
HiDe learns hierarchical control for complex tasks by separating planning and control.
problem Solving long horizon control tasks with generalization to unseen scenarios.
method Functional decomposition of state-action spaces, RL-based planner, modular transfer of policy layers.
result Generalizes across unseen test environments and scales to longer horizons.
Two DRL policies collaborate to solve NP-hard routing problems.
problem Solving complex routing problems like TSP without expert knowledge.
method Learning Collaborative Policies (LCP) using seeder and reviser policies.
result Improves solution quality over single-policy DRL on various NP-hard routing problems.
Hierarchical RL simplifies exploration in RL tasks.
problem Why does hierarchy work well in RL?
method Evaluated hierarchical RL on various tasks, focusing on exploration benefits.
result Most benefits of hierarchy can be attributed to improved exploration.
Novel online algorithm for hierarchical imitation learning.
problem Scalability issue in reinforcement learning and options discovery.
method Online Baum-Welch algorithm for hierarchical imitation learning.
result The online algorithm outperforms the batch version in both discrete and continuous environments.
We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the…
FEN model learns fairness and efficiency in multi-agent systems.
problem Fairness in multi-agent systems for stability and productivity.
method Hierarchical reinforcement learning model with fair-efficient reward and sub-policies.
result FEN model easily learns both fairness and efficiency in multi-agent scenarios.
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcs.LG
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for …
Free Random Projection enhances reinforcement learning by naturally incorporating hierarchical structure.
problem Improving reinforcement learning algorithms for better generalization and adaptability.
method Introduces Free Random Projection, a method that uses free probability theory to create random orthogonal matrices encoding hierarchical structure.
result Empirically shows consistent improvement in generalization over standard methods on multi-environment benchmarks.
Coagent policy gradient algorithms (CPGAs) are reinforcement learning algorithms for training a class of stochastic neural networks called coagent networks. In this work, we prove that CPGAs converge to locally optimal policies. Additionally, we extend prior theory to encompass asynchronous and recurrent coagent networ…
Deep reinforcement learning (DRL) is capable of learning high-performing policies on a variety of complex high-dimensional tasks, ranging from video games to robotic manipulation. However, standard DRL methods often suffer from poor sample efficiency, partially because they aim to be entirely problem-agnostic. In this …
We explore Deep Reinforcement Learning in a parameterized action space. Specifically, we investigate how to achieve sample-efficient end-to-end training in these tasks. We propose a new compact architecture for the tasks where the parameter policy is conditioned on the output of the discrete action policy. We also prop…
Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning. One popular approach for addressing this challenge is the options framework (Sutton et al., 1999). However, only recently in (Bacon et al., 2017) was a policy gradient…
Graph Pointer Networks and hierarchical reinforcement learning solve combinatorial optimization problems like TSP.
problem Traveling Salesman Problem (TSP) with constraints.
method Graph Pointer Networks (GPNs) and hierarchical reinforcement learning.
result GPNs and hierarchical RL find optimal solutions for TSP and TSP with time windows.
This paper investigates generalisation in multi-agent games, where the generality of the agent can be evaluated by playing against opponents it hasn't seen during training. We propose two new games with concealed information and complex, non-transitive reward structure (think rock/paper/scissors). It turns out that mos…
Robotic table tennis learns efficient policies to return balls at 100Hz.
problem Developing efficient robotic table tennis strategies.
method Model-free reinforcement learning using evolutionary search on CNN-based policies.
result Robots can develop multi-modal styles (forehand and backhand) with 80% return rate.
The successful application of general reinforcement learning algorithms to real-world robotics applications is often limited by their high data requirements. We introduce Regularized Hierarchical Policy Optimization (RHPO) to improve data-efficiency for domains with multiple dominant tasks and ultimately reduce require…
Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this p…
Paper tackles RL for power grid topology optimization.
problem Managing large action spaces in growing power networks.
method Hierarchical multi-agent reinforcement learning (MARL) framework.
result MARL framework outperforms single-agent RL methods.
Global convergence proved for multi-agent LQRs with hierarchical actor-critic.
problem Challenges in understanding multi-agent reinforcement learning algorithms.
method Developed a hierarchical actor-critic algorithm for partially exchangeable agents.
result Global linear convergence to optimal policy proved.
We propose a novel framework for multi-task reinforcement learning (MTRL). Using a variational inference formulation, we learn policies that generalize across both changing dynamics and goals. The resulting policies are parametrized by shared parameters that allow for transfer between different dynamics and goal condit…