Large Language Models for Robotics: A Survey

• URL: http://arxiv.org/abs/2311.07226v1
• Date: Mon, 13 Nov 2023 10:46:35 GMT
• Title: Large Language Models for Robotics: A Survey
• Authors: Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, Philip S. Yu
• Abstract summary: Large language models (LLMs) possess the ability to process and generate natural language, facilitating efficient interaction and collaboration with robots. This review aims to summarize the applications of LLMs in robotics, delving into their impact and contributions to key areas such as robot control, perception, decision-making, and path planning.
• Score: 40.76581696885846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The human ability to learn, generalize, and control complex manipulation tasks through multi-modality feedback suggests a unique capability, which we refer to as dexterity intelligence. Understanding and assessing this intelligence is a complex task. Amidst the swift progress and extensive proliferation of large language models (LLMs), their applications in the field of robotics have garnered increasing attention. LLMs possess the ability to process and generate natural language, facilitating efficient interaction and collaboration with robots. Researchers and engineers in the field of robotics have recognized the immense potential of LLMs in enhancing robot intelligence, human-robot interaction, and autonomy. Therefore, this comprehensive review aims to summarize the applications of LLMs in robotics, delving into their impact and contributions to key areas such as robot control, perception, decision-making, and path planning. We first provide an overview of the background and development of LLMs for robotics, followed by a description of the benefits of LLMs for robotics and recent advancements in robotics models based on LLMs. We then delve into the various techniques used in the model, including those employed in perception, decision-making, control, and interaction. Finally, we explore the applications of LLMs in robotics and some potential challenges they may face in the near future. Embodied intelligence is the future of intelligent science, and LLMs-based robotics is one of the promising but challenging paths to achieve this.

Related papers

• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics? [33.573056018368504]
  This study introduces the first benchmark for evaluating Multimodal LLM for Robotic (MMRo) benchmark. We identify four essential capabilities perception, task planning, visual reasoning, and safety measurement that MLLMs must possess to qualify as the robot's central processing unit. Our findings indicate that no single model excels in all areas, suggesting that current MLLMs are not yet trustworthy enough to serve as the cognitive core for robots.
  arXiv  Detail & Related papers  (2024-06-28T07:09:06Z)
• Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration [4.2460673279562755]
  Large Language Models (LLMs) are gaining popularity in the field of robotics. This paper proposes a novel approach to enhance the performance of LLM-based autonomous manipulation through Human-Robot Collaboration (HRC) The approach involves using a prompted GPT-4 language model to decompose high-level language commands into sequences of motions that can be executed by the robot.
  arXiv  Detail & Related papers  (2024-06-20T08:23:49Z)
• Large Language Models for Robotics: Opportunities, Challenges, and Perspectives [46.57277568357048]
  Large language models (LLMs) have undergone significant expansion and have been increasingly integrated across various domains. For embodied tasks, where robots interact with complex environments, text-only LLMs often face challenges due to a lack of compatibility with robotic visual perception. We propose a framework that utilizes multimodal GPT-4V to enhance embodied task planning through the combination of natural language instructions and robot visual perceptions.
  arXiv  Detail & Related papers  (2024-01-09T03:22:16Z)
• WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
  This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system. We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration. We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
  arXiv  Detail & Related papers  (2023-08-30T11:35:21Z)
• Language to Rewards for Robotic Skill Synthesis [37.21434094015743]
  We introduce a new paradigm that harnesses large language models (LLMs) to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions.
  arXiv  Detail & Related papers  (2023-06-14T17:27:10Z)
• LLM as A Robotic Brain: Unifying Egocentric Memory and Control [77.0899374628474]
  Embodied AI focuses on the study and development of intelligent systems that possess a physical or virtual embodiment (i.e. robots) Memory and control are the two essential parts of an embodied system and usually require separate frameworks to model each of them. We propose a novel framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control.
  arXiv  Detail & Related papers  (2023-04-19T00:08:48Z)
• Tiny Robot Learning: Challenges and Directions for Machine Learning in Resource-Constrained Robots [57.27442333662654]
  Machine learning (ML) has become a pervasive tool across computing systems. Tiny robot learning is the deployment of ML on resource-constrained low-cost autonomous robots. Tiny robot learning is subject to challenges from size, weight, area, and power (SWAP) constraints. This paper gives a brief survey of the tiny robot learning space, elaborates on key challenges, and proposes promising opportunities for future work in ML system design.
  arXiv  Detail & Related papers  (2022-05-11T19:36:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.