How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds

• URL: http://arxiv.org/abs/2006.07409v1
• Date: Fri, 12 Jun 2020 18:24:06 GMT
• Title: How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds
• Authors: Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, Mark O. Riedl
• Abstract summary: We introduce Q*BERT, an agent that learns to build a knowledge graph of the world by answering questions. We further introduce MC!Q*BERT an agent that uses a knowledge-graph-based intrinsic motivation to detect bottlenecks. We present an ablation study and results demonstrating how our method outperforms the current state-of-the-art on nine text games.
• Score: 16.626095390308304
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-based games are long puzzles or quests, characterized by a sequence of sparse and potentially deceptive rewards. They provide an ideal platform to develop agents that perceive and act upon the world using a combinatorially sized natural language state-action space. Standard Reinforcement Learning agents are poorly equipped to effectively explore such spaces and often struggle to overcome bottlenecks---states that agents are unable to pass through simply because they do not see the right action sequence enough times to be sufficiently reinforced. We introduce Q*BERT, an agent that learns to build a knowledge graph of the world by answering questions, which leads to greater sample efficiency. To overcome bottlenecks, we further introduce MC!Q*BERT an agent that uses an knowledge-graph-based intrinsic motivation to detect bottlenecks and a novel exploration strategy to efficiently learn a chain of policy modules to overcome them. We present an ablation study and results demonstrating how our method outperforms the current state-of-the-art on nine text games, including the popular game, Zork, where, for the first time, a learning agent gets past the bottleneck where the player is eaten by a Grue.

Related papers

• SPRING: Studying the Paper and Reasoning to Play Games [102.5587155284795]
  We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM) In experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories.
  arXiv  Detail & Related papers  (2023-05-24T18:14:35Z)
• A Minimal Approach for Natural Language Action Space in Text-based Games [103.21433712630953]
  This paper revisits the challenge of exploring the action space in text-based games (TGs) We propose $ epsilon$-admissible exploration, a minimal approach of utilizing admissible actions, for training phase. We present a text-based actor-critic (TAC) agent that produces textual commands for game, solely from game observations.
  arXiv  Detail & Related papers  (2023-05-06T16:05:27Z)
• Long-Term Exploration in Persistent MDPs [68.8204255655161]
  We propose an exploration method called Rollback-Explore (RbExplore) In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge.
  arXiv  Detail & Related papers  (2021-09-21T13:47:04Z)
• Generalization in Text-based Games via Hierarchical Reinforcement Learning [42.70991837415775]
  We introduce a hierarchical framework built upon the knowledge graph-based RL agent. In the high level, a meta-policy is executed to decompose the whole game into a set of subtasks specified by textual goals. In the low level, a sub-policy is executed to conduct goal-conditioned reinforcement learning.
  arXiv  Detail & Related papers  (2021-09-21T05:27:33Z)
• First return, then explore [18.876005532689234]
  Go-Explore is a family of algorithms that explicitly remembers promising states and first returning to such states before intentionally exploring. Go-Explore solves all heretofore unsolved Atari games and surpasses the state of the art on all hard-exploration games. We show that adding a goal-conditioned policy can further improve Go-Explore's exploration efficiency and enable it to handleity throughout training.
  arXiv  Detail & Related papers  (2020-04-27T16:31:26Z)
• Learning Dynamic Belief Graphs to Generalize on Text-Based Games [55.59741414135887]
  Playing text-based games requires skills in processing natural language and sequential decision making. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text.
  arXiv  Detail & Related papers  (2020-02-21T04:38:37Z)
• How To Avoid Being Eaten By a Grue: Exploration Strategies for Text-Adventure Agents [17.215984752298443]
  We introduce two new game state exploration strategies for text-based games. We compare our exploration strategies against strong baselines on the classic text-adventure game, Zork1.
  arXiv  Detail & Related papers  (2020-02-19T17:18:20Z)
• Never Give Up: Learning Directed Exploration Strategies [63.19616370038824]
  We propose a reinforcement learning agent to solve hard exploration games by learning a range of directed exploratory policies. We construct an episodic memory-based intrinsic reward using k-nearest neighbors over the agent's recent experience to train the directed exploratory policies. A self-supervised inverse dynamics model is used to train the embeddings of the nearest neighbour lookup, biasing the novelty signal towards what the agent can control.
  arXiv  Detail & Related papers  (2020-02-14T13:57:22Z)
• Exploration Based Language Learning for Text-Based Games [72.30525050367216]
  This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents.
  arXiv  Detail & Related papers  (2020-01-24T03:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.