Multi-Agent Systems for Robotic Autonomy with LLMs

• URL: http://arxiv.org/abs/2505.05762v1
• Date: Fri, 09 May 2025 03:52:37 GMT
• Title: Multi-Agent Systems for Robotic Autonomy with LLMs
• Authors: Junhong Chen, Ziqi Yang, Haoyuan G Xu, Dandan Zhang, George Mylonas,
• Abstract summary: The framework includes three core agents: Task Analyst, Robot Designer, and Reinforcement Learning Designer.<n>Results demonstrate that the proposed system can design feasible robots with control strategies when appropriate task inputs are provided.
• Score: 7.113794752528622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Since the advent of Large Language Models (LLMs), various research based on such models have maintained significant academic attention and impact, especially in AI and robotics. In this paper, we propose a multi-agent framework with LLMs to construct an integrated system for robotic task analysis, mechanical design, and path generation. The framework includes three core agents: Task Analyst, Robot Designer, and Reinforcement Learning Designer. Outputs are formatted as multimodal results, such as code files or technical reports, for stronger understandability and usability. To evaluate generalizability comparatively, we conducted experiments with models from both GPT and DeepSeek. Results demonstrate that the proposed system can design feasible robots with control strategies when appropriate task inputs are provided, exhibiting substantial potential for enhancing the efficiency and accessibility of robotic system development in research and industrial applications.

Related papers

• An LLM-enabled Multi-Agent Autonomous Mechatronics Design Framework [49.633199780510864]
  This work proposes a multi-agent autonomous mechatronics design framework, integrating expertise across mechanical design, optimization, electronics, and software engineering.<n> operating primarily through a language-driven workflow, the framework incorporates structured human feedback to ensure robust performance under real-world constraints.<n>A fully functional autonomous vessel was developed with optimized propulsion, cost-effective electronics, and advanced control.
  arXiv  Detail & Related papers  (2025-04-20T16:57:45Z)
• Large Language Models as Natural Selector for Embodied Soft Robot Design [5.023206838671049]
  This paper introduces RoboCrafter-QA, a novel benchmark to evaluate whether Large Language Models can learn representations of soft robot designs.<n>Our experiments reveal that while these models exhibit promising capabilities in learning design representations, they struggle with fine-grained distinctions between designs with subtle performance differences.
  arXiv  Detail & Related papers  (2025-03-04T03:55:10Z)
• On the Exploration of LM-Based Soft Modular Robot Design [26.847859137653487]
  Large language models (LLMs) have demonstrated promising capabilities in modeling real-world knowledge. In this paper, we explore the potential of using LLMs to aid in the design of soft modular robots. Our model performs well in evaluations for designing soft modular robots with uni- and bi-directional and stair-descending capabilities.
  arXiv  Detail & Related papers  (2024-11-01T04:03:05Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents [33.77674812074215]
  We introduce a novel multi-agent framework designed to enable effective collaboration among heterogeneous robots.<n>We propose a self-prompted approach, where agents comprehend robot URDF files and call robot kinematics tools to generate descriptions of their physics capabilities.<n>The Habitat-MAS benchmark is designed to assess how a multi-agent framework handles tasks that require embodiment-aware reasoning.
  arXiv  Detail & Related papers  (2024-10-30T03:20:01Z)
• Learning Multi-agent Multi-machine Tending by Mobile Robots [13.812524211429444]
  We introduce a multi-agent multi-machine tending learning framework by mobile robots based on Multi-agent Reinforcement Learning (MARL) techniques.<n>An attention-based encoding mechanism is developed and integrated into Multi-agent Proximal Policy Optimization (MAPPO) algorithm to boost its performance for machine tending scenarios.<n>Our model (AB-MAPPO) outperformed MAPPO in this new challenging scenario in terms of task success, safety, and resources utilization.
  arXiv  Detail & Related papers  (2024-08-29T19:57:52Z)
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
  We present a framework for intuitive robot programming by non-experts. We leverage natural language prompts and contextual information from the Robot Operating System (ROS) Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
  arXiv  Detail & Related papers  (2024-06-28T08:28:38Z)
• LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead [14.834072370183106]
  This paper explores the transformative potential of integrating Large Language Models into Multi-Agent (LMA) systems.<n>By leveraging the collaborative and specialized abilities of multiple agents, LMA systems enable autonomous problem-solving, improve robustness, and provide scalable solutions for managing the complexity of real-world software projects.
  arXiv  Detail & Related papers  (2024-04-07T07:05:40Z)
• An Interactive Agent Foundation Model [49.77861810045509]
  We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
  arXiv  Detail & Related papers  (2024-02-08T18:58:02Z)
• Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis [82.59451639072073]
  General-purpose robots operate seamlessly in any environment, with any object, and utilize various skills to complete diverse tasks. As a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models, we devote this survey to exploring how foundation models can be applied to general-purpose robotics.
  arXiv  Detail & Related papers  (2023-12-14T10:02:55Z)
• Tiny Robot Learning: Challenges and Directions for Machine Learning in Resource-Constrained Robots [57.27442333662654]
  Machine learning (ML) has become a pervasive tool across computing systems. Tiny robot learning is the deployment of ML on resource-constrained low-cost autonomous robots. Tiny robot learning is subject to challenges from size, weight, area, and power (SWAP) constraints. This paper gives a brief survey of the tiny robot learning space, elaborates on key challenges, and proposes promising opportunities for future work in ML system design.
  arXiv  Detail & Related papers  (2022-05-11T19:36:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.