An Autonomous Non-monolithic Agent with Multi-mode Exploration based on Options Framework

• URL: http://arxiv.org/abs/2305.01322v3
• Date: Sat, 4 May 2024 03:16:07 GMT
• Title: An Autonomous Non-monolithic Agent with Multi-mode Exploration based on Options Framework
• Authors: JaeYoon Kim, Junyu Xuan, Christy Liang, Farookh Hussain,
• Abstract summary: Non-monolithic exploration research has emerged to examine the mode-switching exploration behaviour of humans and animals. The ultimate purpose of our research is to enable an agent to decide when to explore or exploit autonomously.
• Score: 2.823645435281551
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most exploration research on reinforcement learning (RL) has paid attention to `the way of exploration', which is `how to explore'. The other exploration research, `when to explore', has not been the main focus of RL exploration research. The issue of `when' of a monolithic exploration in the usual RL exploration behaviour binds an exploratory action to an exploitational action of an agent. Recently, a non-monolithic exploration research has emerged to examine the mode-switching exploration behaviour of humans and animals. The ultimate purpose of our research is to enable an agent to decide when to explore or exploit autonomously. We describe the initial research of an autonomous multi-mode exploration of non-monolithic behaviour in an options framework. The higher performance of our method is shown against the existing non-monolithic exploration method through comparative experimental results.

Related papers

• First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs [2.0690113422225997]
  First-Explore represents a significant step towards developing meta-RL algorithms capable of human-like exploration on a broader range of domains. Our method, First-Explore, overcomes the limitation by learning two policies: one to solely explore, and one to solely exploit.
  arXiv  Detail & Related papers  (2023-07-05T13:20:21Z)
• On the Importance of Exploration for Generalization in Reinforcement Learning [89.63074327328765]
  We propose EDE: Exploration via Distributional Ensemble, a method that encourages exploration of states with high uncertainty. Our algorithm is the first value-based approach to achieve state-of-the-art on both Procgen and Crafter.
  arXiv  Detail & Related papers  (2023-06-08T18:07:02Z)
• First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation [7.021281655855703]
  Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. Key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state. We refer to such exploration after a goal is reached as 'post-exploration'
  arXiv  Detail & Related papers  (2022-12-06T18:56:47Z)
• When to Go, and When to Explore: The Benefit of Post-Exploration in Intrinsic Motivation [7.021281655855703]
  Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. We refer to such exploration after a goal is reached as 'post-exploration' We introduce new methodology to adaptively decide when to post-explore and for how long to post-explore.
  arXiv  Detail & Related papers  (2022-03-29T16:50:12Z)
• Long-Term Exploration in Persistent MDPs [68.8204255655161]
  We propose an exploration method called Rollback-Explore (RbExplore) In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge.
  arXiv  Detail & Related papers  (2021-09-21T13:47:04Z)
• Exploration in Deep Reinforcement Learning: A Comprehensive Survey [24.252352133705735]
  Deep Reinforcement Learning (DRL) and Deep Multi-agent Reinforcement Learning (MARL) have achieved significant success across a wide range of domains, such as game AI, autonomous vehicles, robotics and finance. DRL and deep MARL agents are widely known to be sample-inefficient and millions of interactions are usually needed even for relatively simple game settings. This paper provides a comprehensive survey on existing exploration methods in DRL and deep MARL.
  arXiv  Detail & Related papers  (2021-09-14T13:16:33Z)
• Cooperative Exploration for Multi-Agent Deep Reinforcement Learning [127.4746863307944]
  We propose cooperative multi-agent exploration (CMAE) for deep reinforcement learning. The goal is selected from multiple projected state spaces via a normalized entropy-based technique. We demonstrate that CMAE consistently outperforms baselines on various tasks.
  arXiv  Detail & Related papers  (2021-07-23T20:06:32Z)
• Exploration and Incentives in Reinforcement Learning [107.42240386544633]
  We consider complex exploration problems, where each agent faces the same (but unknown) MDP. Agents control the choice of policies, whereas an algorithm can only issue recommendations. We design an algorithm which explores all reachable states in the MDP.
  arXiv  Detail & Related papers  (2021-02-28T00:15:53Z)
• Reannealing of Decaying Exploration Based On Heuristic Measure in Deep Q-Network [82.20059754270302]
  We propose an algorithm based on the idea of reannealing, that aims at encouraging exploration only when it is needed. We perform an illustrative case study showing that it has potential to both accelerate training and obtain a better policy.
  arXiv  Detail & Related papers  (2020-09-29T20:40:00Z)
• Intrinsic Exploration as Multi-Objective RL [29.124322674133]
  Intrinsic motivation enables reinforcement learning (RL) agents to explore when rewards are very sparse. We propose a framework based on multi-objective RL where both exploration and exploitation are being optimized as separate objectives. This formulation brings the balance between exploration and exploitation at a policy level, resulting in advantages over traditional methods.
  arXiv  Detail & Related papers  (2020-04-06T02:37:29Z)
• Never Give Up: Learning Directed Exploration Strategies [63.19616370038824]
  We propose a reinforcement learning agent to solve hard exploration games by learning a range of directed exploratory policies. We construct an episodic memory-based intrinsic reward using k-nearest neighbors over the agent's recent experience to train the directed exploratory policies. A self-supervised inverse dynamics model is used to train the embeddings of the nearest neighbour lookup, biasing the novelty signal towards what the agent can control.
  arXiv  Detail & Related papers  (2020-02-14T13:57:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.