Towards an Interpretable Hierarchical Agent Framework using Semantic Goals

• URL: http://arxiv.org/abs/2210.08412v1
• Date: Sun, 16 Oct 2022 02:04:13 GMT
• Title: Towards an Interpretable Hierarchical Agent Framework using Semantic Goals
• Authors: Bharat Prakash, Nicholas Waytowich, Tim Oates, Tinoosh Mohsenin
• Abstract summary: This work introduces an interpretable hierarchical agent framework by combining planning and semantic goal directed reinforcement learning. We evaluate our framework on a robotic block manipulation task and show that it performs better than other methods.
• Score: 6.677083312952721
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to solve long horizon temporally extended tasks with reinforcement learning has been a challenge for several years now. We believe that it is important to leverage both the hierarchical structure of complex tasks and to use expert supervision whenever possible to solve such tasks. This work introduces an interpretable hierarchical agent framework by combining planning and semantic goal directed reinforcement learning. We assume access to certain spatial and haptic predicates and construct a simple and powerful semantic goal space. These semantic goal representations are more interpretable, making expert supervision and intervention easier. They also eliminate the need to write complex, dense reward functions thereby reducing human engineering effort. We evaluate our framework on a robotic block manipulation task and show that it performs better than other methods, including both sparse and dense reward functions. We also suggest some next steps and discuss how this framework makes interaction and collaboration with humans easier.

Related papers

• MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint [40.3872201560003]
  Hierarchical reinforcement learning (HRL) uses a hierarchical framework that divides tasks into subgoals and completes them sequentially. Current methods struggle to find suitable subgoals for ensuring a stable learning process. We propose a general hierarchical reinforcement learning framework incorporating human feedback and dynamic distance constraints.
  arXiv  Detail & Related papers  (2024-02-22T03:11:09Z)
• Hierarchical reinforcement learning with natural language subgoals [26.725710518119044]
  We use data from humans solving tasks to softly supervise the goal space for a set of long range tasks in a 3D embodied environment. This has two advantages: first, it is easy to generate this data from naive human participants; second, it is flexible enough to represent a vast range of sub-goals in human-relevant tasks. Our approach outperforms agents that clone expert behavior on these tasks, as well as HRL from scratch without this supervised sub-goal space.
  arXiv  Detail & Related papers  (2023-09-20T18:03:04Z)
• Deep Hierarchical Planning from Pixels [86.14687388689204]
  Director is a method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels.
  arXiv  Detail & Related papers  (2022-06-08T18:20:15Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Divide & Conquer Imitation Learning [75.31752559017978]
  Imitation Learning can be a powerful approach to bootstrap the learning process. We present a novel algorithm designed to imitate complex robotic tasks from the states of an expert trajectory. We show that our method imitates a non-holonomic navigation task and scales to a complex simulated robotic manipulation task with very high sample efficiency.
  arXiv  Detail & Related papers  (2022-04-15T09:56:50Z)
• Possibility Before Utility: Learning And Using Hierarchical Affordances [21.556661319375255]
  Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures. We present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning.
  arXiv  Detail & Related papers  (2022-03-23T19:17:22Z)
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution [54.385344986265714]
  We propose a persistent spatial semantic representation method to bridge the gap between language and robot actions. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.
  arXiv  Detail & Related papers  (2021-07-12T17:47:19Z)
• Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
  Continually solving new, unsolved tasks is the key to learning diverse behaviors. In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. We propose setting up an automatic curriculum for goals that the agent needs to solve. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T03:58:25Z)
• Weakly-Supervised Reinforcement Learning for Controllable Behavior [126.04932929741538]
  Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. In many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. We introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks.
  arXiv  Detail & Related papers  (2020-04-06T17:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.