Curious Hierarchical Actor-Critic Reinforcement Learning

• URL: http://arxiv.org/abs/2005.03420v3
• Date: Mon, 17 Aug 2020 08:45:36 GMT
• Title: Curious Hierarchical Actor-Critic Reinforcement Learning
• Authors: Frank R\"oder, Manfred Eppe, Phuong D.H. Nguyen and Stefan Wermter
• Abstract summary: Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches. We develop a method that combines hierarchical reinforcement learning with curiosity. We demonstrate in several continuous-space environments that curiosity can more than double the learning performance and success rates.
• Score: 13.225264876433528
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches to break down difficult problems into a sequence of simpler ones and to overcome reward sparsity. However, there is a lack of approaches that combine these paradigms, and it is currently unknown whether curiosity also helps to perform the hierarchical abstraction. As a novelty and scientific contribution, we tackle this issue and develop a method that combines hierarchical reinforcement learning with curiosity. Herein, we extend a contemporary hierarchical actor-critic approach with a forward model to develop a hierarchical notion of curiosity. We demonstrate in several continuous-space environments that curiosity can more than double the learning performance and success rates for most of the investigated benchmarking problems. We also provide our source code and a supplementary video.

Related papers

• A Comprehensive Review of Few-shot Action Recognition [64.47305887411275]
  Few-shot action recognition aims to address the high cost and impracticality of manually labeling complex and variable video data. It requires accurately classifying human actions in videos using only a few labeled examples per class.
  arXiv  Detail & Related papers  (2024-07-20T03:53:32Z)
• Cyclophobic Reinforcement Learning [2.2940141855172036]
  In environments with sparse rewards, finding a good inductive bias for exploration is crucial to the agent's success. We propose a new intrinsic reward that is cyclophobic, i.e., it does not reward novelty, but punishes redundancy by avoiding cycles. Augmenting the cyclophobic intrinsic reward with a sequence of hierarchical representations we are able to achieve excellent results in the MiniGrid and MiniHack environments.
  arXiv  Detail & Related papers  (2023-08-30T09:38:44Z)
• A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning [58.107474025048866]
  Forgetting refers to the loss or deterioration of previously acquired knowledge. Forgetting is a prevalent phenomenon observed in various other research domains within deep learning.
  arXiv  Detail & Related papers  (2023-07-16T16:27:58Z)
• Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning [0.6526824510982799]
  In this article, we discuss the limitations of intrinsic curiosity module in sparse-reward multi-agent reinforcement learning. We propose a method called I-Go-Explore that combines the intrinsic curiosity module with the Go-Explore framework to alleviate the detachment problem.
  arXiv  Detail & Related papers  (2023-02-21T17:00:05Z)
• Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning [98.78136504619539]
  Causal Triplet is a causal representation learning benchmark featuring visually more complex scenes. We show that models built with the knowledge of disentangled or object-centric representations significantly outperform their distributed counterparts.
  arXiv  Detail & Related papers  (2023-01-12T17:43:38Z)
• Exploration in Deep Reinforcement Learning: A Survey [4.066140143829243]
  Exploration techniques are of primary importance when solving sparse reward problems. In sparse reward problems, the reward is rare, which means that the agent will not find the reward often by acting randomly. This review provides a comprehensive overview of existing exploration approaches.
  arXiv  Detail & Related papers  (2022-05-02T12:03:44Z)
• On Credit Assignment in Hierarchical Reinforcement Learning [0.0]
  Hierarchical Reinforcement Learning (HRL) has held longstanding promise to advance reinforcement learning. We show how e.g., a 1-step hierarchical backup' can be seen as a conventional multistep backup with $n$ skip connections over time. We develop a new hierarchical algorithm Hier$Q_k(lambda)$, for which we demonstrate that hierarchical credit assignment alone can already boost agent performance.
  arXiv  Detail & Related papers  (2022-03-07T11:13:09Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Towards Open World Object Detection [68.79678648726416]
  ORE: Open World Object Detector is based on contrastive clustering and energy based unknown identification. We find that identifying and characterizing unknown instances helps to reduce confusion in an incremental object detection setting.
  arXiv  Detail & Related papers  (2021-03-03T18:58:18Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• Obstacle Tower Without Human Demonstrations: How Far a Deep Feed-Forward Network Goes with Reinforcement Learning [1.699937048243873]
  The Obstacle Tower Challenge is the task to master a procedurally generated chain of levels that subsequently get harder to complete. We present an approach that performed competitively (placed 7th) but starts completely from scratch by means of Deep Reinforcement Learning.
  arXiv  Detail & Related papers  (2020-04-01T16:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.