Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions

• URL: http://arxiv.org/abs/2305.10783v1
• Date: Thu, 18 May 2023 07:51:33 GMT
• Title: Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions
• Authors: Shrestha Mohanty and Negar Arabzadeh and Julia Kiseleva and Artem Zholus and Milagro Teruel and Ahmed Awadallah and Yuxuan Sun and Kavya Srinet and Arthur Szlam
• Abstract summary: Human intelligence's adaptability is remarkable, allowing us to adjust to new tasks and multi-modal environments swiftly. The research community is actively pursuing the development of interactive "embodied agents" These agents must possess the ability to promptly request feedback in case communication breaks down or instructions are unclear.
• Score: 23.318236094953072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human intelligence's adaptability is remarkable, allowing us to adjust to new tasks and multi-modal environments swiftly. This skill is evident from a young age as we acquire new abilities and solve problems by imitating others or following natural language instructions. The research community is actively pursuing the development of interactive "embodied agents" that can engage in natural conversations with humans and assist them with real-world tasks. These agents must possess the ability to promptly request feedback in case communication breaks down or instructions are unclear. Additionally, they must demonstrate proficiency in learning new vocabulary specific to a given domain. In this paper, we made the following contributions: (1) a crowd-sourcing tool for collecting grounded language instructions; (2) the largest dataset of grounded language instructions; and (3) several state-of-the-art baselines. These contributions are suitable as a foundation for further research.

Related papers

• Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use [16.425032085699698]
  It is desirable for embodied agents to have the ability to leverage human language to gain explicit or implicit knowledge for learning tasks. It's not clear how to incorporate rich language use to facilitate task learning. This paper studies different types of language inputs in facilitating reinforcement learning.
  arXiv  Detail & Related papers  (2024-10-31T17:59:52Z)
• Policy Learning with a Language Bottleneck [65.99843627646018]
  Policy Learning with a Language Bottleneck (PLLBB) is a framework enabling AI agents to generate linguistic rules. PLLBB alternates between a rule generation step guided by language models, and an update step where agents learn new policies guided by rules. In a two-player communication game, a maze solving task, and two image reconstruction tasks, we show thatPLLBB agents are not only able to learn more interpretable and generalizable behaviors, but can also share the learned rules with human users.
  arXiv  Detail & Related papers  (2024-05-07T08:40:21Z)
• Towards More Human-like AI Communication: A Review of Emergent Communication Research [0.0]
  Emergent communication (Emecom) is a field of research aiming to develop artificial agents capable of using natural language. In this review, we delineate all the common proprieties we find across the literature and how they relate to human interactions. We identify two subcategories and highlight their characteristics and open challenges.
  arXiv  Detail & Related papers  (2023-08-01T14:43:10Z)
• Collecting Interactive Multi-modal Datasets for Grounded Language Understanding [66.30648042100123]
  We formalized the collaborative embodied agent using natural language task. We developed a tool for extensive and scalable data collection. We collected the first dataset for interactive grounded language understanding.
  arXiv  Detail & Related papers  (2022-11-12T02:36:32Z)
• IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022 [63.07251290802841]
  We propose IGLU: Interactive Grounded Language Understanding in a Collaborative Environment. The primary goal of the competition is to approach the problem of how to develop interactive embodied agents. This research challenge is naturally related, but not limited, to two fields of study that are highly relevant to the NeurIPS community.
  arXiv  Detail & Related papers  (2022-05-27T06:12:48Z)
• Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021 [58.196738777207315]
  We propose emphIGLU: Interactive Grounded Language Understanding in a Collaborative Environment. The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language instructions in a collaborative environment.
  arXiv  Detail & Related papers  (2022-05-05T01:20:09Z)
• NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment [71.11505407453072]
  We propose IGLU: Interactive Grounded Language Understanding in a Collaborative Environment. The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language instructions in a collaborative environment. This research challenge is naturally related, but not limited, to two fields of study that are highly relevant to the NeurIPS community: Natural Language Understanding and Generation (NLU/G) and Reinforcement Learning (RL)
  arXiv  Detail & Related papers  (2021-10-13T07:13:44Z)
• Few-shot Language Coordination by Modeling Theory of Mind [95.54446989205117]
  We study the task of few-shot $textitlanguage coordination$. We require the lead agent to coordinate with a $textitpopulation$ of agents with different linguistic abilities. This requires the ability to model the partner's beliefs, a vital component of human communication.
  arXiv  Detail & Related papers  (2021-07-12T19:26:11Z)
• Emergent Multi-Agent Communication in the Deep Learning Era [26.764052787245728]
  The ability to cooperate through language is a defining feature of humans. As the perceptual, motory and planning capabilities of deep artificial networks increase, researchers are studying whether they also can develop a shared language to interact.
  arXiv  Detail & Related papers  (2020-06-03T17:50:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.