Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning

• URL: http://arxiv.org/abs/2402.17930v1
• Date: Tue, 27 Feb 2024 23:06:53 GMT
• Title: Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning
• Authors: Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, Joshua B. Tenenbaum
• Abstract summary: This paper introduces cooperative language-guided inverse plan search (CLIPS) Our agent assists a human by modeling them as a cooperative planner who communicates joint plans to the assistant. We evaluate these capabilities in two cooperative planning domains (Doors, Keys & Gems and VirtualHome)
• Score: 52.91457780361305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: People often give instructions whose meaning is ambiguous without further context, expecting that their actions or goals will disambiguate their intentions. How can we build assistive agents that follow such instructions in a flexible, context-sensitive manner? This paper introduces cooperative language-guided inverse plan search (CLIPS), a Bayesian agent architecture for pragmatic instruction following and goal assistance. Our agent assists a human by modeling them as a cooperative planner who communicates joint plans to the assistant, then performs multimodal Bayesian inference over the human's goal from actions and language, using large language models (LLMs) to evaluate the likelihood of an instruction given a hypothesized plan. Given this posterior, our assistant acts to minimize expected goal achievement cost, enabling it to pragmatically follow ambiguous instructions and provide effective assistance even when uncertain about the goal. We evaluate these capabilities in two cooperative planning domains (Doors, Keys & Gems and VirtualHome), finding that CLIPS significantly outperforms GPT-4V, LLM-based literal instruction following and unimodal inverse planning in both accuracy and helpfulness, while closely matching the inferences and assistive judgments provided by human raters.

Related papers

• Infer Human's Intentions Before Following Natural Language Instructions [24.197496779892383]
  We propose a new framework, Follow Instructions with Social and Embodied Reasoning (FISER), aiming for better natural language instruction following in collaborative tasks. Our framework makes explicit inferences about human goals and intentions as intermediate reasoning steps. We empirically demonstrate that using social reasoning to explicitly infer human intentions before making action plans surpasses purely end-to-end approaches.
  arXiv  Detail & Related papers  (2024-09-26T17:19:49Z)
• ThinkBot: Embodied Instruction Following with Thought Chain Reasoning [66.09880459084901]
  Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. We propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions. Our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.
  arXiv  Detail & Related papers  (2023-12-12T08:30:09Z)
• Inferring the Goals of Communicating Agents from Actions and Instructions [47.5816320484482]
  We introduce a model of a cooperative team where one agent, the principal, may communicate natural language instructions about their shared plan to another agent, the assistant. We show how a third person observer can infer the team's goal via multi-modal inverse planning from actions and instructions. We evaluate this approach by comparing it with human goal inferences in a multi-agent gridworld, finding that our model's inferences closely correlate with human judgments.
  arXiv  Detail & Related papers  (2023-06-28T13:43:46Z)
• Safe Explicable Planning [3.3869539907606603]
  We propose Safe Explicable Planning (SEP) to support the specification of a safety bound. Our approach generalizes the consideration of multiple objectives stemming from multiple models. We provide formal proofs that validate the desired theoretical properties of these methods.
  arXiv  Detail & Related papers  (2023-04-04T21:49:02Z)
• NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants [79.27554831580309]
  We study how to build socially intelligent robots to assist people in their homes. We focus on assistance with online goal inference, where robots must simultaneously infer humans' goals.
  arXiv  Detail & Related papers  (2023-01-12T18:59:34Z)
• Neuro-Symbolic Causal Language Planning with Commonsense Prompting [67.06667162430118]
  Language planning aims to implement complex high-level goals by decomposition into simpler low-level steps. Previous methods require either manual exemplars or annotated programs to acquire such ability from large language models. This paper proposes Neuro-Symbolic Causal Language Planner (CLAP) that elicits procedural knowledge from the LLMs with commonsense-infused prompting.
  arXiv  Detail & Related papers  (2022-06-06T22:09:52Z)
• Learning Action Conditions from Instructional Manuals for Instruction Understanding [48.52663250368341]
  We propose a task dubbed action condition inference, and collecting a high-quality, human annotated dataset of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach to automatically construct large-scale training instances from online instructional manuals, and curate a densely human-annotated and validated dataset to study how well the current NLP models can infer action-condition dependencies in instruction texts.
  arXiv  Detail & Related papers  (2022-05-25T00:19:59Z)
• GoalNet: Inferring Conjunctive Goal Predicates from Human Plan Demonstrations for Robot Instruction Following [15.405156791794191]
  Our goal is to enable a robot to learn how to sequence its actions to perform tasks specified as natural language instructions. We introduce a novel neuro-symbolic model, GoalNet, for contextual and task dependent inference of goal predicates. GoalNet demonstrates a significant improvement (51%) in the task completion rate in comparison to a state-of-the-art rule-based approach.
  arXiv  Detail & Related papers  (2022-05-14T15:14:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.