Augmented Reality Demonstrations for Scalable Robot Imitation Learning

• URL: http://arxiv.org/abs/2403.13910v1
• Date: Wed, 20 Mar 2024 18:30:12 GMT
• Title: Augmented Reality Demonstrations for Scalable Robot Imitation Learning
• Authors: Yue Yang, Bryce Ikeda, Gedas Bertasius, Daniel Szafir,
• Abstract summary: This paper presents an innovative solution: an Augmented Reality (AR)-assisted framework for demonstration collection. We empower non-roboticist users to produce demonstrations for robot IL using devices like the HoloLens 2. We validate our approach with experiments on three classical robotics tasks: reach, push, and pick-and-place.
• Score: 25.026589453708347
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robot Imitation Learning (IL) is a widely used method for training robots to perform manipulation tasks that involve mimicking human demonstrations to acquire skills. However, its practicality has been limited due to its requirement that users be trained in operating real robot arms to provide demonstrations. This paper presents an innovative solution: an Augmented Reality (AR)-assisted framework for demonstration collection, empowering non-roboticist users to produce demonstrations for robot IL using devices like the HoloLens 2. Our framework facilitates scalable and diverse demonstration collection for real-world tasks. We validate our approach with experiments on three classical robotics tasks: reach, push, and pick-and-place. The real robot performs each task successfully while replaying demonstrations collected via AR.

Related papers

• VITAL: Visual Teleoperation to Enhance Robot Learning through Human-in-the-Loop Corrections [10.49712834719005]
  We propose a low-cost visual teleoperation system for bimanual manipulation tasks, called VITAL. Our approach leverages affordable hardware and visual processing techniques to collect demonstrations. We enhance the generalizability and robustness of the learned policies by utilizing both real and simulated environments.
  arXiv  Detail & Related papers  (2024-07-30T23:29:47Z)
• Open-TeleVision: Teleoperation with Immersive Active Visual Feedback [17.505318269362512]
  Open-TeleVision allows operators to actively perceive the robot's surroundings in a stereoscopic manner. The system mirrors the operator's arm and hand movements on the robot, creating an immersive experience. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:55:35Z)
• SWBT: Similarity Weighted Behavior Transformer with the Imperfect Demonstration for Robotic Manipulation [32.78083518963342]
  We propose a novel framework named Similarity Weighted Behavior Transformer (SWBT) SWBT effectively learn from both expert and imperfect demonstrations without interaction with environments. We are the first to attempt to integrate imperfect demonstrations into the offline imitation learning setting for robot manipulation tasks.
  arXiv  Detail & Related papers  (2024-01-17T04:15:56Z)
• RoboCLIP: One Demonstration is Enough to Learn Robot Policies [72.24495908759967]
  RoboCLIP is an online imitation learning method that uses a single demonstration in the form of a video demonstration or a textual description of the task to generate rewards. RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need to have the same demonstration and deployment domains.
  arXiv  Detail & Related papers  (2023-10-11T21:10:21Z)
• Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations [66.47064743686953]
  Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation. Videos of humans performing tasks, on the other hand, are much cheaper to collect since they eliminate the need for expertise in robotic teleoperation. In this work, we augment narrow robotic imitation datasets with broad unlabeled human video demonstrations to greatly enhance the generalization of eye-in-hand visuomotor policies.
  arXiv  Detail & Related papers  (2023-07-12T07:04:53Z)
• AR2-D2:Training a Robot Without a Robot [53.10633639596096]
  We introduce AR2-D2, a system for collecting demonstrations which does not require people with specialized training. AR2-D2 is a framework in the form of an iOS app that people can use to record a video of themselves manipulating any object. We show that data collected via our system enables the training of behavior cloning agents in manipulating real objects.
  arXiv  Detail & Related papers  (2023-06-23T23:54:26Z)
• Learning Video-Conditioned Policies for Unseen Manipulation Tasks [83.2240629060453]
  Video-conditioned Policy learning maps human demonstrations of previously unseen tasks to robot manipulation skills. We learn our policy to generate appropriate actions given current scene observations and a video of the target task. We validate our approach on a set of challenging multi-task robot manipulation environments and outperform state of the art.
  arXiv  Detail & Related papers  (2023-05-10T16:25:42Z)
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation [55.31301153979621]
  We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2021-09-28T16:18:54Z)
• Lifelong Robotic Reinforcement Learning by Retaining Experiences [61.79346922421323]
  Many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In this work, we study a practical sequential multi-task RL problem motivated by the practical constraints of physical robotic systems. We derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set.
  arXiv  Detail & Related papers  (2021-09-19T18:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.