Language Guided Exploration for RL Agents in Text Environments

• URL: http://arxiv.org/abs/2403.03141v1
• Date: Tue, 5 Mar 2024 17:26:41 GMT
• Title: Language Guided Exploration for RL Agents in Text Environments
• Authors: Hitesh Golchha, Sahil Yerawar, Dhruvesh Patel, Soham Dan, Keerthiram Murugesan
• Abstract summary: Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. We introduce Language Guided Exploration (LGE) framework, which uses a pre-trained language model to provide decision-level guidance to an RL agent (called EXPLORER) We observe that on ScienceWorld, a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.
• Score: 15.256908785183617
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like $\textit{tabula rasa}$ reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE ) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al.,2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.

Related papers

• Symbolic Learning Enables Self-Evolving Agents [55.625275970720374]
  We introduce agent symbolic learning, a systematic framework that enables language agents to optimize themselves on their own. Agent symbolic learning is designed to optimize the symbolic network within language agents by mimicking two fundamental algorithms in connectionist learning. We conduct proof-of-concept experiments on both standard benchmarks and complex real-world tasks.
  arXiv  Detail & Related papers  (2024-06-26T17:59:18Z)
• EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning [23.83162741035859]
  We present EXPLORER, which is an exploration-guided reasoning agent for textual reinforcement learning. Our experiments show that EXPLORER outperforms the baseline agents on Text-World cooking (TW-Cooking) and Text-World Commonsense (TWC) games.
  arXiv  Detail & Related papers  (2024-03-15T21:22:37Z)
• GLIDE-RL: Grounded Language Instruction through DEmonstration in RL [7.658523833511356]
  Training efficient Reinforcement Learning (RL) agents grounded in natural language has been a long-standing challenge. We present a novel algorithm, Grounded Language Instruction through DEmonstration in RL (GLIDE-RL) that introduces a teacher-instructor-student curriculum learning framework. In this multi-agent framework, the teacher and the student agents learn simultaneously based on the student's current skill level.
  arXiv  Detail & Related papers  (2024-01-03T17:32:13Z)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models [56.25156596019168]
  This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs) Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
  arXiv  Detail & Related papers  (2023-11-30T03:59:31Z)
• ExpeL: LLM Agents Are Experiential Learners [60.54312035818746]
  We introduce the Experiential Learning (ExpeL) agent to allow learning from agent experiences without requiring parametric updates. Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions.
  arXiv  Detail & Related papers  (2023-08-20T03:03:34Z)
• Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling [101.59430768507997]
  Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world. We propose using few-shot large language models (LLMs) to hypothesize an Abstract World Model (AWM) Our method of hypothesizing an AWM with LLMs and then verifying the AWM based on agent experience not only increases sample efficiency over contemporary methods by an order of magnitude.
  arXiv  Detail & Related papers  (2023-01-28T02:04:07Z)
• A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges [38.70863329476517]
  Reinforcement Learning (RL) is a popular machine learning paradigm where intelligent agents interact with the environment to fulfill a long-term goal. Despite the encouraging results achieved, the deep neural network-based backbone is widely deemed as a black box that impedes practitioners to trust and employ trained agents in realistic scenarios where high security and reliability are essential. To alleviate this issue, a large volume of literature devoted to shedding light on the inner workings of the intelligent agents has been proposed, by constructing intrinsic interpretability or post-hoc explainability.
  arXiv  Detail & Related papers  (2022-11-12T13:52:06Z)
• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization [73.74371798168642]
  We introduce an open-source modular library, RL4LMs, for optimizing language generators with reinforcement learning. Next, we present the GRUE benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO, that learns to effectively reduce the action space in language generation.
  arXiv  Detail & Related papers  (2022-10-03T21:38:29Z)
• WordCraft: An Environment for Benchmarking Commonsense Agents [107.20421897619002]
  We propose WordCraft, an RL environment based on Little Alchemy 2. This lightweight environment is fast to run and built upon entities and relations inspired by real-world semantics.
  arXiv  Detail & Related papers  (2020-07-17T18:40:46Z)
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text [12.88819706338837]
  Recent work has described neural-network-based agents that are trained with reinforcement learning to execute language-like commands in simulated worlds. We propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions.
  arXiv  Detail & Related papers  (2020-05-19T12:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.