COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

• URL: http://arxiv.org/abs/2409.15146v2
• Date: Wed, 25 Sep 2024 05:59:08 GMT
• Title: COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models
• Authors: Kehui Liu, Zixin Tang, Dong Wang, Zhigang Wang, Bin Zhao, Xuelong Li,
• Abstract summary: COHERENT is a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems. A Proposal-Execution-Feedback-Adjustment mechanism is designed to decompose and assign actions for individual robots. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency.
• Score: 49.24666980374751
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Leveraging the powerful reasoning capabilities of large language models (LLMs), recent LLM-based robot task planning methods yield promising results. However, they mainly focus on single or multiple homogeneous robots on simple tasks. Practically, complex long-horizon tasks always require collaborations among multiple heterogeneous robots especially with more complex action spaces, which makes these tasks more challenging. To this end, we propose COHERENT, a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems including quadrotors, robotic dogs, and robotic arms. Specifically, a Proposal-Execution-Feedback-Adjustment (PEFA) mechanism is designed to decompose and assign actions for individual robots, where a centralized task assigner makes a task planning proposal to decompose the complex task into subtasks, and then assigns subtasks to robot executors. Each robot executor selects a feasible action to implement the assigned subtask and reports self-reflection feedback to the task assigner for plan adjustment. The PEFA loops until the task is completed. Moreover, we create a challenging heterogeneous multi-robot task planning benchmark encompassing 100 complex long-horizon tasks. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency. The experimental videos, code, and benchmark are released at https://github.com/MrKeee/COHERENT.

Related papers

• A Comparison of Prompt Engineering Techniques for Task Planning and Execution in Service Robotics [16.064583670720587]
  We compare prompt engineering techniques and combinations thereof within the application of high-level task planning and execution in service robotics. We define a diverse set of tasks and a simple set of functionalities in simulation, and measure task completion accuracy and execution time for several state-of-the-art models.
  arXiv  Detail & Related papers  (2024-10-30T13:22:55Z)
• $\textbf{EMOS}$: $\textbf{E}$mbodiment-aware Heterogeneous $\textbf{M}$ulti-robot $\textbf{O}$perating $\textbf{S}$ystem with LLM Agents [33.77674812074215]
  We introduce a novel multi-agent framework designed to enable effective collaboration among heterogeneous robots. We propose a self-prompted approach, where agents comprehend robot URDF files and call robot kinematics tools to generate descriptions of their physics capabilities. The Habitat-MAS benchmark is designed to assess how a multi-agent framework handles tasks that require embodiment-aware reasoning.
  arXiv  Detail & Related papers  (2024-10-30T03:20:01Z)
• Grounding Language Models in Autonomous Loco-manipulation Tasks [3.8363685417355557]
  We propose a novel framework that learns, selects, and plans behaviors based on tasks in different scenarios. We leverage the planning and reasoning features of the large language model (LLM), constructing a hierarchical task graph. Experiments in simulation and real-world using the CENTAURO robot show that the language model based planner can efficiently adapt to new loco-manipulation tasks.
  arXiv  Detail & Related papers  (2024-09-02T15:27:48Z)
• Safe Task Planning for Language-Instructed Multi-Robot Systems using Conformal Prediction [11.614036749291216]
  We introduce a new distributed multi-robot planner, S-ATLAS for Safe plAnning for Teams of Language-instructed AgentS, that is capable of achieving user-defined mission success rates. We show, both theoretically and empirically, that the proposed planner can achieve user-specified task success rates while minimizing the overall number of help requests.
  arXiv  Detail & Related papers  (2024-02-23T15:02:44Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• LEMMA: Learning Language-Conditioned Multi-Robot Manipulation [21.75163634731677]
  LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) LeMMA features 8 types of procedurally generated tasks with varying degree of complexity. For each task, we provide 800 expert demonstrations and human instructions for training and evaluations.
  arXiv  Detail & Related papers  (2023-08-02T04:37:07Z)
• AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation [50.737355245505334]
  We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks. The resulting dataset AlphaBlock consists of 35 comprehensive high-level tasks of multi-step text plans and paired observation.
  arXiv  Detail & Related papers  (2023-05-30T09:54:20Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• Lifelong Robotic Reinforcement Learning by Retaining Experiences [61.79346922421323]
  Many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In this work, we study a practical sequential multi-task RL problem motivated by the practical constraints of physical robotic systems. We derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set.
  arXiv  Detail & Related papers  (2021-09-19T18:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.