ChatGPT for Robotics: Design Principles and Model Abilities
- URL: http://arxiv.org/abs/2306.17582v2
- Date: Wed, 19 Jul 2023 19:30:28 GMT
- Title: ChatGPT for Robotics: Design Principles and Model Abilities
- Authors: Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor
- Abstract summary: We outline a strategy that combines design principles for prompt engineering and the creation of a high-level function library.
We focus our evaluations on the effectiveness of different prompt engineering techniques and dialog strategies towards the execution of various types of robotics tasks.
Our study encompasses a range of tasks within the robotics domain, from basic logical, geometrical, and mathematical reasoning all the way to complex domains such as aerial navigation, manipulation, and embodied agents.
- Score: 25.032064314822243
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper presents an experimental study regarding the use of OpenAI's
ChatGPT for robotics applications. We outline a strategy that combines design
principles for prompt engineering and the creation of a high-level function
library which allows ChatGPT to adapt to different robotics tasks, simulators,
and form factors. We focus our evaluations on the effectiveness of different
prompt engineering techniques and dialog strategies towards the execution of
various types of robotics tasks. We explore ChatGPT's ability to use free-form
dialog, parse XML tags, and to synthesize code, in addition to the use of
task-specific prompting functions and closed-loop reasoning through dialogues.
Our study encompasses a range of tasks within the robotics domain, from basic
logical, geometrical, and mathematical reasoning all the way to complex domains
such as aerial navigation, manipulation, and embodied agents. We show that
ChatGPT can be effective at solving several of such tasks, while allowing users
to interact with it primarily via natural language instructions. In addition to
these studies, we introduce an open-sourced research tool called PromptCraft,
which contains a platform where researchers can collaboratively upload and vote
on examples of good prompting schemes for robotics applications, as well as a
sample robotics simulator with ChatGPT integration, making it easier for users
to get started with using ChatGPT for robotics.
Related papers
- $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge.
We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
arXiv Detail & Related papers (2024-10-31T17:22:30Z) - RoboScript: Code Generation for Free-Form Manipulation Tasks across Real
and Simulation [77.41969287400977]
This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation.
We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language.
We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
arXiv Detail & Related papers (2024-02-22T15:12:00Z) - Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming [4.779196219827508]
We introduce a novel framework for a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting.
This system allows users to interact with robot agents through natural language, each powered by individual GPT cores.
A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a multi-robot environment.
arXiv Detail & Related papers (2023-12-12T12:26:48Z) - Creative Robot Tool Use with Large Language Models [47.11935262923095]
This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning.
We develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments.
arXiv Detail & Related papers (2023-10-19T18:02:15Z) - WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system.
We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration.
We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
arXiv Detail & Related papers (2023-08-30T11:35:21Z) - AutoML-GPT: Automatic Machine Learning with GPT [74.30699827690596]
We propose developing task-oriented prompts and automatically utilizing large language models (LLMs) to automate the training pipeline.
We present the AutoML-GPT, which employs GPT as the bridge to diverse AI models and dynamically trains models with optimized hyper parameters.
This approach achieves remarkable results in computer vision, natural language processing, and other challenging areas.
arXiv Detail & Related papers (2023-05-04T02:09:43Z) - Logic programming for deliberative robotic task planning [2.610470075814367]
We present a survey on recent advances in the application of logic programming to the problem of task planning.
We analyze different planners and their suitability for specific robotic applications, based on expressivity in domain representation, computational efficiency and software implementation.
arXiv Detail & Related papers (2023-01-18T14:11:55Z) - ProgPrompt: Generating Situated Robot Task Plans using Large Language
Models [68.57918965060787]
Large language models (LLMs) can be used to score potential next actions during task planning.
We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
arXiv Detail & Related papers (2022-09-22T20:29:49Z) - iRoPro: An interactive Robot Programming Framework [2.7651063843287718]
iRoPro allows users with little to no technical background to teach a robot new reusable actions.
We implement iRoPro as an end-to-end system on a Baxter Research Robot.
arXiv Detail & Related papers (2021-12-08T13:53:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.