Large Language Models for Human-Robot Interaction: Opportunities and Risks

• URL: http://arxiv.org/abs/2405.00693v1
• Date: Tue, 26 Mar 2024 15:36:40 GMT
• Title: Large Language Models for Human-Robot Interaction: Opportunities and Risks
• Authors: Jesse Atuhurra,
• Abstract summary: We present a meta-study about the potential of large language models if deployed in social robots. We study how these language models could be safely trained to understand'' societal norms and issues. We hope this study provides a resourceful guide to other robotics researchers interested in incorporating language models in their robots.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The tremendous development in large language models (LLM) has led to a new wave of innovations and applications and yielded research results that were initially forecast to take longer. In this work, we tap into these recent developments and present a meta-study about the potential of large language models if deployed in social robots. We place particular emphasis on the applications of social robots: education, healthcare, and entertainment. Before being deployed in social robots, we also study how these language models could be safely trained to ``understand'' societal norms and issues, such as trust, bias, ethics, cognition, and teamwork. We hope this study provides a resourceful guide to other robotics researchers interested in incorporating language models in their robots.

Related papers

• Plan-and-Act using Large Language Models for Interactive Agreement [8.07285448283823]
  Recent large language models (LLMs) are capable of planning robot actions. Key problem of applying LLMs in situational HRI is balancing between "respecting the current human's activity" and "prioritizing the robot's task"
  arXiv  Detail & Related papers  (2025-04-01T23:41:05Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics? [33.573056018368504]
  This study introduces the first benchmark for evaluating Multimodal LLM for Robotic (MMRo) benchmark. We identify four essential capabilities perception, task planning, visual reasoning, and safety measurement that MLLMs must possess to qualify as the robot's central processing unit. Our findings indicate that no single model excels in all areas, suggesting that current MLLMs are not yet trustworthy enough to serve as the cognitive core for robots.
  arXiv  Detail & Related papers  (2024-06-28T07:09:06Z)
• Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration [4.2460673279562755]
  Large Language Models (LLMs) are gaining popularity in the field of robotics. This paper proposes a novel approach to enhance the performance of LLM-based autonomous manipulation through Human-Robot Collaboration (HRC) The approach involves using a prompted GPT-4 language model to decompose high-level language commands into sequences of motions that can be executed by the robot.
  arXiv  Detail & Related papers  (2024-06-20T08:23:49Z)
• Large Language Models for Robotics: Opportunities, Challenges, and Perspectives [46.57277568357048]
  Large language models (LLMs) have undergone significant expansion and have been increasingly integrated across various domains. For embodied tasks, where robots interact with complex environments, text-only LLMs often face challenges due to a lack of compatibility with robotic visual perception. We propose a framework that utilizes multimodal GPT-4V to enhance embodied task planning through the combination of natural language instructions and robot visual perceptions.
  arXiv  Detail & Related papers  (2024-01-09T03:22:16Z)
• Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis [82.59451639072073]
  General-purpose robots operate seamlessly in any environment, with any object, and utilize various skills to complete diverse tasks. As a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models, we devote this survey to exploring how foundation models can be applied to general-purpose robotics.
  arXiv  Detail & Related papers  (2023-12-14T10:02:55Z)
• Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming [4.779196219827508]
  We introduce a novel framework for a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with robot agents through natural language, each powered by individual GPT cores. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a multi-robot environment.
  arXiv  Detail & Related papers  (2023-12-12T12:26:48Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• Large Language Models for Robotics: A Survey [40.76581696885846]
  Large language models (LLMs) possess the ability to process and generate natural language, facilitating efficient interaction and collaboration with robots. This review aims to summarize the applications of LLMs in robotics, delving into their impact and contributions to key areas such as robot control, perception, decision-making, and path planning.
  arXiv  Detail & Related papers  (2023-11-13T10:46:35Z)
• WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
  This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system. We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration. We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
  arXiv  Detail & Related papers  (2023-08-30T11:35:21Z)
• LLM as A Robotic Brain: Unifying Egocentric Memory and Control [77.0899374628474]
  Embodied AI focuses on the study and development of intelligent systems that possess a physical or virtual embodiment (i.e. robots) Memory and control are the two essential parts of an embodied system and usually require separate frameworks to model each of them. We propose a novel framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control.
  arXiv  Detail & Related papers  (2023-04-19T00:08:48Z)
• Open-World Object Manipulation using Pre-trained Vision-Language Models [72.87306011500084]
  For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary. We develop a simple approach, which leverages a pre-trained vision-language model to extract object-identifying information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments.
  arXiv  Detail & Related papers  (2023-03-02T01:55:10Z)
• World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges [51.92834011423463]
  We focus on the two concepts of world models and predictive coding. In neuroscience, predictive coding proposes that the brain continuously predicts its inputs and adapts to model its own dynamics and control behavior in its environment.
  arXiv  Detail & Related papers  (2023-01-14T06:38:14Z)
• Understanding Natural Language in Context [13.112390442564442]
  We focus on cognitive robots, which have some knowledge-based models of the world and operate by reasoning and planning with this model. Our goal in this research is to translate natural language utterances into this robot's formalism. We do so by combining off-the-shelf SOTA language models, planning tools, and the robot's knowledge-base for better communication.
  arXiv  Detail & Related papers  (2022-05-25T11:52:16Z)
• Semantics for Robotic Mapping, Perception and Interaction: A Survey [93.93587844202534]
  Study of understanding dictates what does the world "mean" to a robot. With humans and robots increasingly operating in the same world, the prospects of human-robot interaction also bring semantics into the picture. Driven by need, as well as by enablers like increasing availability of training data and computational resources, semantics is a rapidly growing research area in robotics.
  arXiv  Detail & Related papers  (2021-01-02T12:34:39Z)
• Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshop [0.819605661841562]
  Meeting human needs requires addressing new challenges in speech technology and user experience design. More powerful adaptation methods are needed, without extensive re-engineering or the collection of massive training data. Since robots operate in real time, their speech and language processing components must also.
  arXiv  Detail & Related papers  (2020-11-11T03:45:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.