Causality-driven Hierarchical Structure Discovery for Reinforcement Learning

• URL: http://arxiv.org/abs/2210.06964v1
• Date: Thu, 13 Oct 2022 12:42:48 GMT
• Title: Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
• Authors: Shaohui Peng, Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen, Xishan Zhang, Zidong Du, Ling Li, Qi Guo, Yunji Chen
• Abstract summary: We propose CDHRL, a causality-driven hierarchical reinforcement learning framework. We show that CDHRL significantly boosts exploration efficiency with the causality-driven paradigm. The results in two complex environments, 2D-Minecraft and Eden, show that CDHRL significantly boosts exploration efficiency with the causality-driven paradigm.
• Score: 36.03953383550469
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hierarchical reinforcement learning (HRL) effectively improves agents' exploration efficiency on tasks with sparse reward, with the guide of high-quality hierarchical structures (e.g., subgoals or options). However, how to automatically discover high-quality hierarchical structures is still a great challenge. Previous HRL methods can hardly discover the hierarchical structures in complex environments due to the low exploration efficiency by exploiting the randomness-driven exploration paradigm. To address this issue, we propose CDHRL, a causality-driven hierarchical reinforcement learning framework, leveraging a causality-driven discovery instead of a randomness-driven exploration to effectively build high-quality hierarchical structures in complicated environments. The key insight is that the causalities among environment variables are naturally fit for modeling reachable subgoals and their dependencies and can perfectly guide to build high-quality hierarchical structures. The results in two complex environments, 2D-Minecraft and Eden, show that CDHRL significantly boosts exploration efficiency with the causality-driven paradigm.

Related papers

• StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization [94.31508613367296]
  Retrieval-augmented generation (RAG) is a key means to effectively enhance large language models (LLMs) We propose StructRAG, which can identify the optimal structure type for the task at hand, reconstruct original documents into this structured format, and infer answers based on the resulting structure. Experiments show that StructRAG achieves state-of-the-art performance, particularly excelling in challenging scenarios.
  arXiv  Detail & Related papers  (2024-10-11T13:52:44Z)
• A Survey on Neural Architecture Search Based on Reinforcement Learning [0.0]
  This paper introduces the overall development of Neural Architecture Search. We then focus mainly on providing an overall and understandable survey about Neural Architecture Search works.
  arXiv  Detail & Related papers  (2024-09-26T17:28:10Z)
• Variable-Agnostic Causal Exploration for Reinforcement Learning [56.52768265734155]
  We introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL) Our approach automatically identifies crucial observation-action steps associated with key variables using attention mechanisms. It constructs the causal graph connecting these steps, which guides the agent towards observation-action pairs with greater causal influence on task completion.
  arXiv  Detail & Related papers  (2024-07-17T09:45:27Z)
• PEAR: Primitive enabled Adaptive Relabeling for boosting Hierarchical Reinforcement Learning [25.84621883831624]
  Hierarchical reinforcement learning has the potential to solve complex long horizon tasks using temporal abstraction and increased exploration. We present primitive enabled adaptive relabeling (PEAR) We first perform adaptive relabeling on a few expert demonstrations to generate efficient subgoal supervision. We then jointly optimize HRL agents by employing reinforcement learning (RL) and imitation learning (IL)
  arXiv  Detail & Related papers  (2023-06-10T09:41:30Z)
• Hierarchical Reinforcement Learning in Complex 3D Environments [16.16652618709808]
  Hierarchical Reinforcement Learning (HRL) agents have the potential to demonstrate appealing capabilities. Recent successes with HRL across different domains provide evidence that practical, effective HRL agents are possible. Despite these successes, visually complex partially observable 3D environments remained a challenge for HRL agents.
  arXiv  Detail & Related papers  (2023-02-28T09:56:36Z)
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning [64.97599673479678]
  We present HIerarchical reinforcement learning Guided by Landmarks (HIGL) HIGL is a novel framework for training a high-level policy with a reduced action space guided by landmarks. Our experiments demonstrate that our framework outperforms prior-arts across a variety of control tasks.
  arXiv  Detail & Related papers  (2021-10-26T12:16:19Z)
• Provable Hierarchy-Based Meta-Reinforcement Learning [50.17896588738377]
  We analyze HRL in the meta-RL setting, where learner learns latent hierarchical structure during meta-training for use in a downstream task. We provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
  arXiv  Detail & Related papers  (2021-10-18T17:56:02Z)
• Hierarchical community structure in networks [1.9981375888949475]
  We present a theoretical study on hierarchical community structure in networks. We address the following questions: 1) How should we define a hierarchy of communities? 2) How do we determine if there is sufficient evidence of a hierarchical structure in a network?
  arXiv  Detail & Related papers  (2020-09-15T16:18:26Z)
• Hierarchical Reinforcement Learning as a Model of Human Task Interleaving [60.95424607008241]
  We develop a hierarchical model of supervisory control driven by reinforcement learning. The model reproduces known empirical effects of task interleaving. The results support hierarchical RL as a plausible model of task interleaving.
  arXiv  Detail & Related papers  (2020-01-04T17:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.