Automatic Robotic Development through Collaborative Framework by Large
Language Models
- URL: http://arxiv.org/abs/2402.03699v2
- Date: Fri, 16 Feb 2024 12:46:03 GMT
- Title: Automatic Robotic Development through Collaborative Framework by Large
Language Models
- Authors: Zhirong Luan and Yujun Lai, Rundong Huang, Xiaruiqi Lan, Liangjun
Chen, Badong Chen
- Abstract summary: We propose an innovative automated collaboration framework inspired by real-world robot developers.
This framework employs multiple LLMs in distinct roles analysts, programmers, and testers.
Analysts delve deep into user requirements, enabling programmers to produce precise code, while testers fine-tune the parameters.
- Score: 13.957351735394683
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite the remarkable code generation abilities of large language models
LLMs, they still face challenges in complex task handling. Robot development, a
highly intricate field, inherently demands human involvement in task allocation
and collaborative teamwork . To enhance robot development, we propose an
innovative automated collaboration framework inspired by real-world robot
developers. This framework employs multiple LLMs in distinct roles analysts,
programmers, and testers. Analysts delve deep into user requirements, enabling
programmers to produce precise code, while testers fine-tune the parameters
based on user feedback for practical robot application. Each LLM tackles
diverse, critical tasks within the development process. Clear collaboration
rules emulate real world teamwork among LLMs. Analysts, programmers, and
testers form a cohesive team overseeing strategy, code, and parameter
adjustments . Through this framework, we achieve complex robot development
without requiring specialized knowledge, relying solely on non experts
participation.
Related papers
- ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
We present a framework for intuitive robot programming by non-experts.
We leverage natural language prompts and contextual information from the Robot Operating System (ROS)
Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
arXiv Detail & Related papers (2024-06-28T08:28:38Z) - LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application [4.519544934630495]
Large Language Models (LLM) and Vision Language Models (VLM) enable robots to ground natural language prompts into control actions.
We propose Language-driven Intention Tracking (LIT) to model the human user's long-term behavior and to predict the next human intention to guide the robot for proactive collaboration.
arXiv Detail & Related papers (2024-06-19T19:18:40Z) - RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Models [49.23588578549434]
Large Language Models (LLMs) have improved the prospects for robotic tasks.
Existing benchmarks are still limited to single tasks with limited generalization capabilities.
We introduce a comprehensive benchmark and an autonomous learning framework, RoboCoder.
arXiv Detail & Related papers (2024-06-06T05:41:47Z) - Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World [13.005764902339523]
We design a blocks-world environment where two agents, each having unique goals and skills, build a target structure together.
To complete the goals, they can act in the world and communicate in natural language.
We adopt chain-of-thought prompts that include intermediate reasoning steps to model the partner's state and identify and correct execution errors.
arXiv Detail & Related papers (2024-03-30T04:48:38Z) - RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis [102.1876259853457]
We propose a tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX.
RoboCodeX decomposes high-level human instructions into multiple object-centric manipulation units consisting of physical preferences such as affordance and safety constraints.
To further enhance the capability to map conceptual and perceptual understanding into control commands, a specialized multimodal reasoning dataset is collected for pre-training and an iterative self-updating methodology is introduced for supervised fine-tuning.
arXiv Detail & Related papers (2024-02-25T15:31:43Z) - RoboScript: Code Generation for Free-Form Manipulation Tasks across Real
and Simulation [77.41969287400977]
This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation.
We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language.
We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
arXiv Detail & Related papers (2024-02-22T15:12:00Z) - Large Language Models for Robotics: Opportunities, Challenges, and
Perspectives [46.57277568357048]
Large language models (LLMs) have undergone significant expansion and have been increasingly integrated across various domains.
For embodied tasks, where robots interact with complex environments, text-only LLMs often face challenges due to a lack of compatibility with robotic visual perception.
We propose a framework that utilizes multimodal GPT-4V to enhance embodied task planning through the combination of natural language instructions and robot visual perceptions.
arXiv Detail & Related papers (2024-01-09T03:22:16Z) - Large Language Models for Robotics: A Survey [40.76581696885846]
Large language models (LLMs) possess the ability to process and generate natural language, facilitating efficient interaction and collaboration with robots.
This review aims to summarize the applications of LLMs in robotics, delving into their impact and contributions to key areas such as robot control, perception, decision-making, and path planning.
arXiv Detail & Related papers (2023-11-13T10:46:35Z) - RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation [68.70755196744533]
RoboGen is a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation.
Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics.
arXiv Detail & Related papers (2023-11-02T17:59:21Z) - Language to Rewards for Robotic Skill Synthesis [37.21434094015743]
We introduce a new paradigm that harnesses large language models (LLMs) to define reward parameters that can be optimized and accomplish variety of robotic tasks.
Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions.
arXiv Detail & Related papers (2023-06-14T17:27:10Z) - Self-collaboration Code Generation via ChatGPT [35.88318116340547]
Large Language Models (LLMs) have demonstrated remarkable code-generation ability, but struggle with complex tasks.
We present a self-collaboration framework for code generation employing LLMs, exemplified by ChatGPT.
To effectively organize and manage this virtual team, we incorporate software-development methodology into the framework.
arXiv Detail & Related papers (2023-04-15T16:33:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.