Learning by Asking for Embodied Visual Navigation and Task Completion

• URL: http://arxiv.org/abs/2302.04865v1
• Date: Thu, 9 Feb 2023 18:59:41 GMT
• Title: Learning by Asking for Embodied Visual Navigation and Task Completion
• Authors: Ying Shen and Ismini Lourentzou
• Abstract summary: We propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. Experimental results show that ELBA achieves improved task performance compared to baseline models without question-answering capabilities.
• Score: 20.0182240268864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Despite recent progress on related vision-language benchmarks, most prior work has focused on building agents that follow instructions rather than endowing agents the ability to ask questions to actively resolve ambiguities arising naturally in embodied environments. To empower embodied agents with the ability to interact with humans, in this work, we propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. We evaluate our model on the TEACH vision-dialog navigation and task completion dataset. Experimental results show that ELBA achieves improved task performance compared to baseline models without question-answering capabilities.

Related papers

• A Survey on Complex Tasks for Goal-Directed Interactive Agents [60.53915548970061]
  This survey compiles relevant tasks and environments for evaluating goal-directed interactive agents. An up-to-date compilation of relevant resources can be found on our project website.
  arXiv  Detail & Related papers  (2024-09-27T08:17:53Z)
• Improving Agent Interactions in Virtual Environments with Language Models [0.9790236766474201]
  This research focuses on a collective building assignment in the Minecraft dataset. We employ language modeling to enhance task understanding through state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-08T06:34:11Z)
• Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? [62.59699229202307]
  Despite advances in AI, it remains a significant challenge to develop interactive task guidance systems. We created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance.
  arXiv  Detail & Related papers  (2023-11-01T15:13:49Z)
• Asking Before Acting: Gather Information in Embodied Decision Making with Language Models [20.282749796376063]
  We show that Large Language Models (LLMs) encounter challenges in efficiently gathering essential information in unfamiliar environments. We propose textitAsking Before Acting (ABA), a method that empowers the agent to proactively inquire with external sources for pertinent information using natural language. We conduct extensive experiments involving a spectrum of environments including text-based household everyday tasks, robot arm manipulation tasks, and real world open domain image based embodied tasks.
  arXiv  Detail & Related papers  (2023-05-25T04:05:08Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Hierarchical Few-Shot Imitation with Skill Transition Models [66.81252581083199]
  Few-shot Imitation with Skill Transition Models (FIST) is an algorithm that extracts skills from offline data and utilizes them to generalize to unseen tasks. We show that FIST is capable of generalizing to new tasks and substantially outperforms prior baselines in navigation experiments.
  arXiv  Detail & Related papers  (2021-07-19T15:56:01Z)
• Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment [73.9469267445146]
  First-person object-interaction tasks in high-fidelity, 3D, simulated environments such as the AI2Thor pose significant sample-efficiency challenges for reinforcement learning agents. We show that one can learn object-interaction tasks from scratch without supervision by learning an attentive object-model as an auxiliary task.
  arXiv  Detail & Related papers  (2020-10-28T19:27:26Z)
• LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities [119.88381048477854]
  We introduce the LEMMA dataset to provide a single home to address missing dimensions with meticulously designed settings. We densely annotate the atomic-actions with human-object interactions to provide ground-truths of the compositionality, scheduling, and assignment of daily activities. We hope this effort would drive the machine vision community to examine goal-directed human activities and further study the task scheduling and assignment in the real world.
  arXiv  Detail & Related papers  (2020-07-31T00:13:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.