Towards Generalist Robots: A Promising Paradigm via Generative
Simulation
- URL: http://arxiv.org/abs/2305.10455v3
- Date: Wed, 30 Aug 2023 00:05:26 GMT
- Title: Towards Generalist Robots: A Promising Paradigm via Generative
Simulation
- Authors: Zhou Xian, Theophile Gervet, Zhenjia Xu, Yi-Ling Qiao, Tsun-Hsuan
Wang, Yian Wang
- Abstract summary: This document serves as a position paper that outlines the authors' vision for a potential pathway towards generalist robots.
The authors believe the proposed paradigm is a feasible path towards accomplishing the long-standing goal of robotics research.
- Score: 18.704506851738365
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This document serves as a position paper that outlines the authors' vision
for a potential pathway towards generalist robots. The purpose of this document
is to share the excitement of the authors with the community and highlight a
promising research direction in robotics and AI. The authors believe the
proposed paradigm is a feasible path towards accomplishing the long-standing
goal of robotics research: deploying robots, or embodied AI agents more
broadly, in various non-factory real-world settings to perform diverse tasks.
This document presents a specific idea for mining knowledge in the latest
large-scale foundation models for robotics research. Instead of directly using
or adapting these models to produce low-level policies and actions, it
advocates for a fully automated generative pipeline (termed as generative
simulation), which uses these models to generate diversified tasks, scenes and
training supervisions at scale, thereby scaling up low-level skill learning and
ultimately leading to a foundation model for robotics that empowers generalist
robots. The authors are actively pursuing this direction, but in the meantime,
they recognize that the ambitious goal of building generalist robots with
large-scale policy training demands significant resources such as computing
power and hardware, and research groups in academia alone may face severe
resource constraints in implementing the entire vision. Therefore, the authors
believe sharing their thoughts at this early stage could foster discussions,
attract interest towards the proposed pathway and related topics from industry
groups, and potentially spur significant technical advancements in the field.
Related papers
- Contextual Affordances for Safe Exploration in Robotic Scenarios [1.7647943747248804]
This paper explores the use of contextual affordances to enable safe exploration and learning in robotic scenarios targeted at the home.
We propose a simple state representation that allows us to extend contextual affordances to larger state spaces.
In the long term, this work could be the foundation for future explorations of human-robot interactions in complex domestic environments.
arXiv Detail & Related papers (2024-05-10T12:12:38Z) - Position Paper: Agent AI Towards a Holistic Intelligence [53.35971598180146]
We emphasize developing Agent AI -- an embodied system that integrates large foundation models into agent actions.
In this paper, we propose a novel large action model to achieve embodied intelligent behavior, the Agent Foundation Model.
arXiv Detail & Related papers (2024-02-28T16:09:56Z) - RoboScript: Code Generation for Free-Form Manipulation Tasks across Real
and Simulation [77.41969287400977]
This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation.
We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language.
We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
arXiv Detail & Related papers (2024-02-22T15:12:00Z) - A Survey on Robotics with Foundation Models: toward Embodied AI [30.999414445286757]
Recent advances in computer vision, natural language processing, and multi-modality learning have shown that the foundation models have superhuman capabilities for specific tasks.
This survey aims to provide a comprehensive and up-to-date overview of foundation models in robotics, focusing on autonomous manipulation and encompassing high-level planning and low-level control.
arXiv Detail & Related papers (2024-02-04T07:55:01Z) - Toward General-Purpose Robots via Foundation Models: A Survey and
Meta-Analysis [73.89558418030418]
Most existing robotic systems have been designed for specific tasks, trained on specific datasets, and deployed within specific environments.
Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models, we devote this survey to exploring how foundation models can be applied to robotics.
arXiv Detail & Related papers (2023-12-14T10:02:55Z) - RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation [68.70755196744533]
RoboGen is a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation.
Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics.
arXiv Detail & Related papers (2023-11-02T17:59:21Z) - A Capability and Skill Model for Heterogeneous Autonomous Robots [69.50862982117127]
capability modeling is considered a promising approach to semantically model functions provided by different machines.
This contribution investigates how to apply and extend capability models from manufacturing to the field of autonomous robots.
arXiv Detail & Related papers (2022-09-22T10:13:55Z) - Can Foundation Models Perform Zero-Shot Task Specification For Robot
Manipulation? [54.442692221567796]
Task specification is critical for engagement of non-expert end-users and adoption of personalized robots.
A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene.
In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
arXiv Detail & Related papers (2022-04-23T19:39:49Z) - Heterogeneous Learning from Demonstration [0.0]
We propose a framework for learning from heterogeneous demonstration based upon Bayesian inference.
We evaluate a suite of approaches on a real-world dataset of gameplay from StarCraft II.
arXiv Detail & Related papers (2020-01-27T03:08:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.