ELLA: Exploration through Learned Language Abstraction

• URL: http://arxiv.org/abs/2103.05825v1
• Date: Wed, 10 Mar 2021 02:18:46 GMT
• Title: ELLA: Exploration through Learned Language Abstraction
• Authors: Suvir Mirchandani, Siddharth Karamcheti, Dorsa Sadigh
• Abstract summary: ELLA is a reward shaping approach that correlates high-level instructions with simpler low-level instructions to enrich the sparse rewards afforded by the environment. ELLA shows a significant gain in sample efficiency across several environments compared to competitive language-based reward shaping and no-shaping methods.
• Score: 6.809870486883877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Building agents capable of understanding language instructions is critical to effective and robust human-AI collaboration. Recent work focuses on training these instruction following agents via reinforcement learning in environments with synthetic language; however, these instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. To this end, we introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach that correlates high-level instructions with simpler low-level instructions to enrich the sparse rewards afforded by the environment. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success on high-level tasks. We learn the termination classifier offline from pairs of instructions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions to low-level instructions. On a suite of complex grid world environments with varying instruction complexities and reward sparsity, ELLA shows a significant gain in sample efficiency across several environments compared to competitive language-based reward shaping and no-shaping methods.

Related papers

• Words as Beacons: Guiding RL Agents with High-Level Language Prompts [6.7236795813629]
  Large Language Models (LLMs) as "teachers" guide the agent's learning process by decomposing complex tasks into subgoals. LLMs can provide subgoals to accomplish the task defined for the environment in a similar fashion to how a human would do. It is possible to query the LLM only during the training phase, enabling agents to operate within the environment without any LLM intervention.
  arXiv  Detail & Related papers  (2024-10-11T08:54:45Z)
• Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments [42.06453257292203]
  We propose a hierarchical framework that combines the deep language comprehension of large language models with the adaptive action-execution capabilities of reinforcement learning agents. We have demonstrated the effectiveness of our approach in two different environments: in IGLU, where agents are instructed to build structures, and in Crafter, where agents perform tasks and interact with objects in the surrounding environment according to language commands.
  arXiv  Detail & Related papers  (2024-07-12T14:19:36Z)
• kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning [50.40636157214161]
  Task-Oriented Parsing (TOP) enables conversational assistants to interpret user commands expressed in natural language. LLMs have achieved impressive performance in computer programs based on a natural language prompt. This paper focuses on harnessing the capabilities of LLMs for semantic parsing tasks.
  arXiv  Detail & Related papers  (2023-12-17T17:26:50Z)
• Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation [172.15808300686584]
  We describe an approach that learns the two tasks simultaneously and exploits their intrinsic correlations to boost the training of each. Our approach improves the performance of various follower models and produces accurate navigation instructions.
  arXiv  Detail & Related papers  (2022-03-30T18:15:26Z)
• LISA: Learning Interpretable Skill Abstractions from Language [85.20587800593293]
  We propose a hierarchical imitation learning framework that can learn diverse, interpretable skills from language-conditioned demonstrations. Our method demonstrates a more natural way to condition on language in sequential decision-making problems.
  arXiv  Detail & Related papers  (2022-02-28T19:43:24Z)
• FILM: Following Instructions in Language with Modular Methods [109.73082108379936]
  Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. We propose a modular method with structured representations that builds a semantic map of the scene and performs exploration with a semantic search policy. Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance.
  arXiv  Detail & Related papers  (2021-10-12T16:40:01Z)
• Skill Induction and Planning with Latent Language [94.55783888325165]
  We formulate a generative model of action sequences in which goals generate sequences of high-level subtask descriptions. We describe how to train this model using primarily unannotated demonstrations by parsing demonstrations into sequences of named high-level subtasks. In trained models, the space of natural language commands indexes a library of skills; agents can use these skills to plan by generating high-level instruction sequences tailored to novel goals.
  arXiv  Detail & Related papers  (2021-10-04T15:36:32Z)
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning [97.10875695679499]
  We propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks.
  arXiv  Detail & Related papers  (2020-12-30T03:35:22Z)
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning [32.82030512053361]
  We propose the use of step-by-step human demonstrations in the form of natural language instructions and action trajectories. We find that human demonstrations help solve the most complex tasks. We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting.
  arXiv  Detail & Related papers  (2020-11-01T14:39:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.