Integrating Disambiguation and User Preferences into Large Language Models for Robot Motion Planning

• URL: http://arxiv.org/abs/2404.14547v1
• Date: Mon, 22 Apr 2024 19:38:37 GMT
• Title: Integrating Disambiguation and User Preferences into Large Language Models for Robot Motion Planning
• Authors: Mohammed Abugurain, Shinkyu Park,
• Abstract summary: framework can interpret humans' navigation commands containing temporal elements and translate their natural language instructions into robot motion planning. We propose methods to resolve the ambiguity in natural language instructions and capture user preferences.
• Score: 1.9912315834033756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a framework that can interpret humans' navigation commands containing temporal elements and directly translate their natural language instructions into robot motion planning. Central to our framework is utilizing Large Language Models (LLMs). To enhance the reliability of LLMs in the framework and improve user experience, we propose methods to resolve the ambiguity in natural language instructions and capture user preferences. The process begins with an ambiguity classifier, identifying potential uncertainties in the instructions. Ambiguous statements trigger a GPT-4-based mechanism that generates clarifying questions, incorporating user responses for disambiguation. Also, the framework assesses and records user preferences for non-ambiguous instructions, enhancing future interactions. The last part of this process is the translation of disambiguated instructions into a robot motion plan using Linear Temporal Logic. This paper details the development of this framework and the evaluation of its performance in various test scenarios.

Related papers

• In-Context Learning Enables Robot Action Prediction in LLMs [52.285739178561705]
  We introduce RoboPrompt, a framework that enables offthe-shelf text-only Large Language Models to directly predict robot actions. Our approach firstally identifiess that capture important moments from an episode. We extract end-effector actions as well as the estimated initial object poses, and both are converted into textual descriptions. This enables an LLM to directly predict robot actions at test time.
  arXiv  Detail & Related papers  (2024-10-16T17:56:49Z)
• Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation [8.931633531104021]
  SAS (Spatially-Aware Speaker) is an instruction generator that uses both structural and semantic knowledge of the environment to produce richer instructions. Our method outperforms existing instruction generation models, evaluated using standard metrics.
  arXiv  Detail & Related papers  (2024-09-09T13:12:11Z)
• Object-Centric Instruction Augmentation for Robotic Manipulation [29.491990994901666]
  We introduce the textitObject-Centric Instruction Augmentation (OCI) framework to augment highly semantic and information-dense language instruction with position cues. We utilize a Multi-modal Large Language Model (MLLM) to weave knowledge of object locations into natural language instruction. We demonstrate that robotic manipulator imitation policies trained with our enhanced instructions outperform those relying solely on traditional language instructions.
  arXiv  Detail & Related papers  (2024-01-05T13:54:45Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• Interpreting User Requests in the Context of Natural Language Standing Instructions [89.12540932734476]
  We develop NLSI, a language-to-program dataset consisting of over 2.4K dialogues spanning 17 domains. A key challenge in NLSI is to identify which subset of the standing instructions is applicable to a given dialogue.
  arXiv  Detail & Related papers  (2023-11-16T11:19:26Z)
• Dialogue-based generation of self-driving simulation scenarios using Large Language Models [14.86435467709869]
  Simulation is an invaluable tool for developing and evaluating controllers for self-driving cars. Current simulation frameworks are driven by highly-specialist domain specific languages. There is often a gap between a concise English utterance and the executable code that captures the user's intent.
  arXiv  Detail & Related papers  (2023-10-26T13:07:01Z)
• CARTIER: Cartographic lAnguage Reasoning Targeted at Instruction Execution for Robots [9.393951367344894]
  This work explores the capacity of large language models to address problems at the intersection of spatial planning and natural language interfaces for navigation. We focus on following complex instructions that are more akin to natural conversation than traditional explicit procedural directives typically seen in robotics. We leverage the 3D simulator AI2Thor to create household query scenarios at scale, and augment it by adding complex language queries for 40 object types.
  arXiv  Detail & Related papers  (2023-07-21T19:09:37Z)
• Query Understanding in the Age of Large Language Models [6.630482733703617]
  We describe a generic framework for interactive query-rewriting using large-language models (LLM) A key aspect of our framework is the ability of the rewriter to fully specify the machine intent by the search engine in natural language. We detail the concept, backed by initial experiments, along with open questions for this interactive query understanding framework.
  arXiv  Detail & Related papers  (2023-06-28T08:24:14Z)
• Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model [63.66204449776262]
  Instruct2Act is a framework that maps multi-modal instructions to sequential actions for robotic manipulation tasks. Our approach is adjustable and flexible in accommodating various instruction modalities and input types. Our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks.
  arXiv  Detail & Related papers  (2023-05-18T17:59:49Z)
• PADL: Language-Directed Physics-Based Character Control [66.517142635815]
  We present PADL, which allows users to issue natural language commands for specifying high-level tasks and low-level skills that a character should perform. We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.
  arXiv  Detail & Related papers  (2023-01-31T18:59:22Z)
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution [54.385344986265714]
  We propose a persistent spatial semantic representation method to bridge the gap between language and robot actions. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.
  arXiv  Detail & Related papers  (2021-07-12T17:47:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.