TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation

• URL: http://arxiv.org/abs/2403.07869v2
• Date: Thu, 21 Mar 2024 19:57:46 GMT
• Title: TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation
• Authors: Shivin Dass, Wensi Ai, Yuqian Jiang, Samik Singh, Jiaheng Hu, Ruohan Zhang, Peter Stone, Ben Abbatematteo, Roberto Martín-Martín,
• Abstract summary: In this work, we demonstrate TeleMoMa, a general and modular interface for whole-body teleoperation of mobile manipulators. TeleMoMa unifies multiple human interfaces including RGB and depth cameras, virtual reality controllers, keyboard, joysticks, etc.
• Score: 38.187217710937084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A critical bottleneck limiting imitation learning in robotics is the lack of data. This problem is more severe in mobile manipulation, where collecting demonstrations is harder than in stationary manipulation due to the lack of available and easy-to-use teleoperation interfaces. In this work, we demonstrate TeleMoMa, a general and modular interface for whole-body teleoperation of mobile manipulators. TeleMoMa unifies multiple human interfaces including RGB and depth cameras, virtual reality controllers, keyboard, joysticks, etc., and any combination thereof. In its more accessible version, TeleMoMa works using simply vision (e.g., an RGB-D camera), lowering the entry bar for humans to provide mobile manipulation demonstrations. We demonstrate the versatility of TeleMoMa by teleoperating several existing mobile manipulators - PAL Tiago++, Toyota HSR, and Fetch - in simulation and the real world. We demonstrate the quality of the demonstrations collected with TeleMoMa by training imitation learning policies for mobile manipulation tasks involving synchronized whole-body motion. Finally, we also show that TeleMoMa's teleoperation channel enables teleoperation on site, looking at the robot, or remote, sending commands and observations through a computer network, and perform user studies to evaluate how easy it is for novice users to learn to collect demonstrations with different combinations of human interfaces enabled by our system. We hope TeleMoMa becomes a helpful tool for the community enabling researchers to collect whole-body mobile manipulation demonstrations. For more information and video results, https://robin-lab.cs.utexas.edu/telemoma-web.

Related papers

• Zero-Cost Whole-Body Teleoperation for Mobile Manipulation [8.71539730969424]
  MoMa-Teleop is a novel teleoperation method that delegates the base motions to a reinforcement learning agent. We demonstrate that our approach results in a significant reduction in task completion time across a variety of robots and tasks.
  arXiv  Detail & Related papers  (2024-09-23T15:09:45Z)
• Open-TeleVision: Teleoperation with Immersive Active Visual Feedback [17.505318269362512]
  Open-TeleVision allows operators to actively perceive the robot's surroundings in a stereoscopic manner. The system mirrors the operator's arm and hand movements on the robot, creating an immersive experience. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:55:35Z)
• Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation [59.21899709023333]
  We develop a system for imitating mobile manipulation tasks that are bimanual and require whole-body control. Mobile ALOHA is a low-cost and whole-body teleoperation system for data collection. Co-training can increase success rates by up to 90%, allowing Mobile ALOHA to autonomously complete complex mobile manipulation tasks.
  arXiv  Detail & Related papers  (2024-01-04T07:55:53Z)
• Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations [66.47064743686953]
  Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation. Videos of humans performing tasks, on the other hand, are much cheaper to collect since they eliminate the need for expertise in robotic teleoperation. In this work, we augment narrow robotic imitation datasets with broad unlabeled human video demonstrations to greatly enhance the generalization of eye-in-hand visuomotor policies.
  arXiv  Detail & Related papers  (2023-07-12T07:04:53Z)
• From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation [26.738893736520364]
  We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. We construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-26T17:59:51Z)
• Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation [54.31414116478024]
  In mobile manipulation (MM), robots can both navigate within and interact with their environment. In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks.
  arXiv  Detail & Related papers  (2021-12-09T23:54:59Z)
• Learning Multi-Arm Manipulation Through Collaborative Teleoperation [63.35924708783826]
  Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks. Many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. We present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms.
  arXiv  Detail & Related papers  (2020-12-12T05:43:43Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)
• A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU [25.451864296962288]
  We present a novel vision-based hand pose regression network (Transteleop) and an IMU-based arm tracking method. Transteleop observes the human hand through a low-cost depth camera and generates depth images of paired robot hand poses. A wearable camera holder enables simultaneous hand-arm control and facilitates the mobility of the whole teleoperation system.
  arXiv  Detail & Related papers  (2020-03-11T10:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.