Human-in-the-Loop Task and Motion Planning for Imitation Learning
- URL: http://arxiv.org/abs/2310.16014v1
- Date: Tue, 24 Oct 2023 17:15:16 GMT
- Title: Human-in-the-Loop Task and Motion Planning for Imitation Learning
- Authors: Ajay Mandlekar, Caelan Garrett, Danfei Xu, Dieter Fox
- Abstract summary: Imitation learning from human demonstrations can teach robots complex manipulation skills, but is time-consuming and labor intensive.
In contrast, Task and Motion Planning (TAMP) systems are automated and excel at solving long-horizon tasks.
We present Human-in-the-Loop Task and Motion Planning (HITL-TAMP), a novel system that leverages the benefits of both approaches.
- Score: 37.75197145733193
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Imitation learning from human demonstrations can teach robots complex
manipulation skills, but is time-consuming and labor intensive. In contrast,
Task and Motion Planning (TAMP) systems are automated and excel at solving
long-horizon tasks, but they are difficult to apply to contact-rich tasks. In
this paper, we present Human-in-the-Loop Task and Motion Planning (HITL-TAMP),
a novel system that leverages the benefits of both approaches. The system
employs a TAMP-gated control mechanism, which selectively gives and takes
control to and from a human teleoperator. This enables the human teleoperator
to manage a fleet of robots, maximizing data collection efficiency. The
collected human data is then combined with an imitation learning framework to
train a TAMP-gated policy, leading to superior performance compared to training
on full task demonstrations. We compared HITL-TAMP to a conventional
teleoperation system -- users gathered more than 3x the number of demos given
the same time budget. Furthermore, proficient agents (75\%+ success) could be
trained from just 10 minutes of non-expert teleoperation data. Finally, we
collected 2.1K demos with HITL-TAMP across 12 contact-rich, long-horizon tasks
and show that the system often produces near-perfect agents. Videos and
additional results at https://hitltamp.github.io .
Related papers
- Open-TeleVision: Teleoperation with Immersive Active Visual Feedback [17.505318269362512]
Open-TeleVision allows operators to actively perceive the robot's surroundings in a stereoscopic manner.
The system mirrors the operator's arm and hand movements on the robot, creating an immersive experience.
We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks.
arXiv Detail & Related papers (2024-07-01T17:55:35Z) - Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition [48.65867987106428]
We introduce a novel system for joint learning between human operators and robots.
It enables human operators to share control of a robot end-effector with a learned assistive agent.
It also allows the human operator to adjust the control ratio to achieve a trade-off between manual and automated control.
arXiv Detail & Related papers (2024-06-29T03:37:29Z) - GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robot [27.410618312830497]
In this paper, we propose GeRM (Generalist Robotic Model)
We utilize offline reinforcement learning to optimize data utilization strategies.
We employ a transformer-based VLA network to process multi-modal inputs and output actions.
arXiv Detail & Related papers (2024-03-20T07:36:43Z) - MimicGen: A Data Generation System for Scalable Robot Learning using
Human Demonstrations [55.549956643032836]
MimicGen is a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations.
We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks.
arXiv Detail & Related papers (2023-10-26T17:17:31Z) - Imitating Task and Motion Planning with Visuomotor Transformers [71.41938181838124]
Task and Motion Planning (TAMP) can autonomously generate large-scale datasets of diverse demonstrations.
In this work, we show that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for robot manipulation.
We present a novel imitation learning system called OPTIMUS that trains large-scale visuomotor Transformer policies by imitating a TAMP agent.
arXiv Detail & Related papers (2023-05-25T17:58:14Z) - PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection [19.04536551595612]
Policy Assisted TeleOperation (PATO) is a system which automates part of the demonstration collection process using a learned assistive policy.
PATO autonomously executes repetitive behaviors in data collection and asks for human input only when it is uncertain about which subtask or behavior to execute.
arXiv Detail & Related papers (2022-12-09T07:38:09Z) - Learning to Coordinate for a Worker-Station Multi-robot System in Planar
Coverage Tasks [16.323122275188354]
We focus on the multi-robot coverage path planning problem in large-scale planar areas with random dynamic interferers.
We introduce a worker-station MRS consisting of multiple workers with limited resources for actual work, and one station with enough resources for resource replenishment.
We propose an end-to-end decentralized online planning method, which simultaneously solves coverage planning for workers and rendezvous planning for station.
arXiv Detail & Related papers (2022-08-05T05:36:42Z) - Error-Aware Imitation Learning from Teleoperation Data for Mobile
Manipulation [54.31414116478024]
In mobile manipulation (MM), robots can both navigate within and interact with their environment.
In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks.
arXiv Detail & Related papers (2021-12-09T23:54:59Z) - Bottom-Up Skill Discovery from Unsegmented Demonstrations for
Long-Horizon Robot Manipulation [55.31301153979621]
We tackle real-world long-horizon robot manipulation tasks through skill discovery.
We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations.
Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks.
arXiv Detail & Related papers (2021-09-28T16:18:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.