Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models

• URL: http://arxiv.org/abs/2405.15143v4
• Date: Fri, 07 Feb 2025 11:10:39 GMT
• Title: Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models
• Authors: Cong Lu, Shengran Hu, Jeff Clune,
• Abstract summary: Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems. We propose Intelligent Go-Explore (IGE) which greatly extends the scope of the original Go-Explore. IGE has a human-like ability to instinctively identify how interesting or promising any new state is.
• Score: 5.404186221463082
• License:
• Abstract: Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems built on the principle of archiving discovered states, and iteratively returning to and exploring from the most promising states. This approach has led to superhuman performance across a wide variety of challenging problems including Atari games and robotic control, but requires manually designing heuristics to guide exploration (i.e., determine which states to save and explore from, and what actions to consider next), which is time-consuming and infeasible in general. To resolve this, we propose Intelligent Go-Explore (IGE) which greatly extends the scope of the original Go-Explore by replacing these handcrafted heuristics with the intelligence and internalized human notions of interestingness captured by giant pretrained foundation models (FMs). This provides IGE with a human-like ability to instinctively identify how interesting or promising any new state is (e.g., discovering new objects, locations, or behaviors), even in complex environments where heuristics are hard to define. Moreover, IGE offers the exciting opportunity to recognize and capitalize on serendipitous discoveries -- states encountered during exploration that are valuable in terms of exploration, yet where what makes them interesting was not anticipated by the human user. We evaluate our algorithm on a diverse range of language and vision-based tasks that require search and exploration. Across these tasks, IGE strongly exceeds classic reinforcement learning and graph search baselines, and also succeeds where prior state-of-the-art FM agents like Reflexion completely fail. Overall, Intelligent Go-Explore combines the tremendous strengths of FMs and the powerful Go-Explore algorithm, opening up a new frontier of research into creating more generally capable agents with impressive exploration capabilities.

Related papers

• MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization [91.80034860399677]
  Reinforcement learning algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. We introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits.
  arXiv  Detail & Related papers  (2024-12-16T18:59:53Z)
• A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond [84.95530356322621]
  This survey presents a systematic review of the advancements in code intelligence. It covers over 50 representative models and their variants, more than 20 categories of tasks, and an extensive coverage of over 680 related works. Building on our examination of the developmental trajectories, we further investigate the emerging synergies between code intelligence and broader machine intelligence.
  arXiv  Detail & Related papers  (2024-03-21T08:54:56Z)
• Time-Myopic Go-Explore: Learning A State Representation for the Go-Explore Paradigm [0.5156484100374059]
  We introduce a novel time-myopic state representation that clusters temporal close states together. We demonstrate the first learned state representation that reliably estimates novelty instead of using the hand-crafted representation. Our approach is evaluated on the hard exploration environments MontezumaRevenge, Gravitar and Frostbite (Atari)
  arXiv  Detail & Related papers  (2023-01-13T16:13:44Z)
• First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation [7.021281655855703]
  Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. Key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state. We refer to such exploration after a goal is reached as 'post-exploration'
  arXiv  Detail & Related papers  (2022-12-06T18:56:47Z)
• When to Go, and When to Explore: The Benefit of Post-Exploration in Intrinsic Motivation [7.021281655855703]
  Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. We refer to such exploration after a goal is reached as 'post-exploration' We introduce new methodology to adaptively decide when to post-explore and for how long to post-explore.
  arXiv  Detail & Related papers  (2022-03-29T16:50:12Z)
• Long-Term Exploration in Persistent MDPs [68.8204255655161]
  We propose an exploration method called Rollback-Explore (RbExplore) In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge.
  arXiv  Detail & Related papers  (2021-09-21T13:47:04Z)
• Cooperative Exploration for Multi-Agent Deep Reinforcement Learning [127.4746863307944]
  We propose cooperative multi-agent exploration (CMAE) for deep reinforcement learning. The goal is selected from multiple projected state spaces via a normalized entropy-based technique. We demonstrate that CMAE consistently outperforms baselines on various tasks.
  arXiv  Detail & Related papers  (2021-07-23T20:06:32Z)
• Batch Exploration with Examples for Scalable Robotic Reinforcement Learning [63.552788688544254]
  Batch Exploration with Examples (BEE) explores relevant regions of the state-space guided by a modest number of human provided images of important states. BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot.
  arXiv  Detail & Related papers  (2020-10-22T17:49:25Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• First return, then explore [18.876005532689234]
  Go-Explore is a family of algorithms that explicitly remembers promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games. We show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handleity throughout training.
  arXiv  Detail & Related papers  (2020-04-27T16:31:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.