Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis

• URL: http://arxiv.org/abs/2309.07675v2
• Date: Wed, 22 Nov 2023 10:24:26 GMT
• Title: Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis
• Authors: Mehdi Zadem and Sergio Mover and Sao Mai Nguyen
• Abstract summary: We introduce a Feudal HRL algorithm that concurrently learns both the goal representation and a hierarchical policy. We evaluate our approach on complex navigation tasks, showing the learned representation is interpretable, transferrable and results in data efficient learning.
• Score: 0.5409704301731713
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Open-ended learning benefits immensely from the use of symbolic methods for goal representation as they offer ways to structure knowledge for efficient and transferable learning. However, the existing Hierarchical Reinforcement Learning (HRL) approaches relying on symbolic reasoning are often limited as they require a manual goal representation. The challenge in autonomously discovering a symbolic goal representation is that it must preserve critical information, such as the environment dynamics. In this paper, we propose a developmental mechanism for goal discovery via an emergent representation that abstracts (i.e., groups together) sets of environment states that have similar roles in the task. We introduce a Feudal HRL algorithm that concurrently learns both the goal representation and a hierarchical policy. The algorithm uses symbolic reachability analysis for neural networks to approximate the transition relation among sets of states and to refine the goal representation. We evaluate our approach on complex navigation tasks, showing the learned representation is interpretable, transferrable and results in data efficient learning.

Related papers

• VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning [86.59849798539312]
  We present Neuro-Symbolic Predicates, a first-order abstraction language that combines the strengths of symbolic and neural knowledge representations. We show that our approach offers better sample complexity, stronger out-of-distribution generalization, and improved interpretability.
  arXiv  Detail & Related papers  (2024-10-30T16:11:05Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• A Novel Neural-symbolic System under Statistical Relational Learning [50.747658038910565]
  We propose a general bi-level probabilistic graphical reasoning framework called GBPGR. In GBPGR, the results of symbolic reasoning are utilized to refine and correct the predictions made by the deep learning models. Our approach achieves high performance and exhibits effective generalization in both transductive and inductive tasks.
  arXiv  Detail & Related papers  (2023-09-16T09:15:37Z)
• Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis [0.0]
  We propose a developmental mechanism for subgoal discovery via an emergent representation that abstracts (i.e., groups together) sets of environment states. We create a HRL algorithm that gradually learns this representation along with the policies and evaluate it on navigation tasks to show the learned representation is interpretable and results in data efficiency.
  arXiv  Detail & Related papers  (2023-09-12T06:53:11Z)
• Improving Deep Representation Learning via Auxiliary Learnable Target Coding [69.79343510578877]
  This paper introduces a novel learnable target coding as an auxiliary regularization of deep representation learning. Specifically, a margin-based triplet loss and a correlation consistency loss on the proposed target codes are designed to encourage more discriminative representations.
  arXiv  Detail & Related papers  (2023-05-30T01:38:54Z)
• Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search [63.3745291252038]
  We propose DiffSES, a novel symbolic learning approach that discovers discrete symbolic policies. By using object-level abstractions instead of raw pixel-level inputs, DiffSES is able to leverage the simplicity and scalability advantages of symbolic expressions. Our experiments demonstrate that DiffSES is able to generate symbolic policies that are simpler and more scalable than state-of-the-art symbolic RL methods.
  arXiv  Detail & Related papers  (2022-12-30T17:50:54Z)
• Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning [17.58129740811116]
  We propose a reward learning approach, Graph-based Equivalence Mappings (GEM) GEM represents a spatial goal specification by a reward function conditioned on i) a graph indicating important spatial relationships between objects and ii) state equivalence mappings for each edge in the graph. We show that GEM can drastically improve the generalizability of the learned goal representations over strong baselines.
  arXiv  Detail & Related papers  (2022-11-24T18:59:06Z)
• Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning [99.38163119531745]
  We show that applying a discretizing bottleneck can improve performance in goal-conditioned RL setups. We experimentally prove the expected return on out-of-distribution goals, while still allowing for specifying goals with expressive structure.
  arXiv  Detail & Related papers  (2022-11-01T03:31:43Z)
• Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning [15.698612710580447]
  We propose a skill learning framework DR-GRL that aims to improve the sample efficiency and policy generalization. In a weakly supervised manner, we propose a Spatial Transform AutoEncoder (STAE) to learn an interpretable and controllable representation. We empirically demonstrate that DR-GRL significantly outperforms the previous methods in sample efficiency and policy generalization.
  arXiv  Detail & Related papers  (2022-02-28T09:05:14Z)
• Learning Neural-Symbolic Descriptive Planning Models via Cube-Space Priors: The Voyage Home (to STRIPS) [13.141761152863868]
  We show that our neuro-symbolic architecture is trained end-to-end to produce a succinct and effective discrete state transition model from images alone. Our target representation is already in a form that off-the-shelf solvers can consume, and opens the door to the rich array of modern search capabilities.
  arXiv  Detail & Related papers  (2020-04-27T15:01:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.