Learning Robot Manipulation from Cross-Morphology Demonstration

• URL: http://arxiv.org/abs/2304.03833v2
• Date: Mon, 30 Oct 2023 02:13:12 GMT
• Title: Learning Robot Manipulation from Cross-Morphology Demonstration
• Authors: Gautam Salhotra, I-Chun Arthur Liu, Gaurav Sukhatme
• Abstract summary: Some Learning from Demonstrations (LfD) methods handle small mismatches in the action spaces of the teacher and student. Here we address the case where the teacher's morphology is substantially different from that of the student. Our framework, Morphological Adaptation in Imitation Learning (MAIL), bridges this gap allowing us to train an agent from demonstrations by other agents with significantly different morphologies.
• Score: 0.9615284569035419
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Some Learning from Demonstrations (LfD) methods handle small mismatches in the action spaces of the teacher and student. Here we address the case where the teacher's morphology is substantially different from that of the student. Our framework, Morphological Adaptation in Imitation Learning (MAIL), bridges this gap allowing us to train an agent from demonstrations by other agents with significantly different morphologies. MAIL learns from suboptimal demonstrations, so long as they provide $\textit{some}$ guidance towards a desired solution. We demonstrate MAIL on manipulation tasks with rigid and deformable objects including 3D cloth manipulation interacting with rigid obstacles. We train a visual control policy for a robot with one end-effector using demonstrations from a simulated agent with two end-effectors. MAIL shows up to $24\%$ improvement in a normalized performance metric over LfD and non-LfD baselines. It is deployed to a real Franka Panda robot, handles multiple variations in properties for objects (size, rotation, translation), and cloth-specific properties (color, thickness, size, material). An overview is on https://uscresl.github.io/mail .

Related papers

• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)
• One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformers [11.875194596371484]
  Humans can learn to complete tasks, even complex ones, after only seeing one or two demonstrations. Our work seeks to emulate this ability, using behavior cloning to learn a task given only a single human demonstration. We develop a novel addition to the temporal ensembling method used by action chunking agents during inference.
  arXiv  Detail & Related papers  (2023-09-18T21:50:26Z)
• One-shot Imitation Learning via Interaction Warping [32.5466340846254]
  We propose a new method, Interaction Warping, for learning SE(3) robotic manipulation policies from a single demonstration. We infer the 3D mesh of each object in the environment using shape warping, a technique for aligning point clouds across object instances. We show successful one-shot imitation learning on three simulated and real-world object re-arrangement tasks.
  arXiv  Detail & Related papers  (2023-06-21T17:26:11Z)
• Imitating Task and Motion Planning with Visuomotor Transformers [71.41938181838124]
  Task and Motion Planning (TAMP) can autonomously generate large-scale datasets of diverse demonstrations. In this work, we show that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for robot manipulation. We present a novel imitation learning system called OPTIMUS that trains large-scale visuomotor Transformer policies by imitating a TAMP agent.
  arXiv  Detail & Related papers  (2023-05-25T17:58:14Z)
• Cross-Domain Transfer via Semantic Skill Imitation [49.83150463391275]
  We propose an approach for semantic imitation, which uses demonstrations from a source domain, e.g. human videos, to accelerate reinforcement learning (RL) Instead of imitating low-level actions like joint velocities, our approach imitates the sequence of demonstrated semantic skills like "opening the microwave" or "turning on the stove"
  arXiv  Detail & Related papers  (2022-12-14T18:46:14Z)
• Multi-Modal Few-Shot Temporal Action Detection [157.96194484236483]
  Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection to new classes. We introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD.
  arXiv  Detail & Related papers  (2022-11-27T18:13:05Z)
• VIMA: General Robot Manipulation with Multimodal Prompts [82.01214865117637]
  We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts. We develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively.
  arXiv  Detail & Related papers  (2022-10-06T17:50:11Z)
• Learning Generalizable Dexterous Manipulation from Human Grasp Affordance [11.060931225148936]
  Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. Recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning. We propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category.
  arXiv  Detail & Related papers  (2022-04-05T16:26:22Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.