Learning Object Manipulation Skills via Approximate State Estimation from Real Videos

• URL: http://arxiv.org/abs/2011.06813v1
• Date: Fri, 13 Nov 2020 08:53:47 GMT
• Title: Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
• Authors: Vladim\'ir Petr\'ik, Makarand Tapaswi, Ivan Laptev, Josef Sivic
• Abstract summary: Humans are adept at learning new tasks by watching a few instructional videos. On the other hand, robots that learn new actions either require a lot of effort through trial and error, or use expert demonstrations that are challenging to obtain. In this paper, we explore a method that facilitates learning object manipulation skills directly from videos.
• Score: 47.958512470724926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans are adept at learning new tasks by watching a few instructional videos. On the other hand, robots that learn new actions either require a lot of effort through trial and error, or use expert demonstrations that are challenging to obtain. In this paper, we explore a method that facilitates learning object manipulation skills directly from videos. Leveraging recent advances in 2D visual recognition and differentiable rendering, we develop an optimization based method to estimate a coarse 3D state representation for the hand and the manipulated object(s) without requiring any supervision. We use these trajectories as dense rewards for an agent that learns to mimic them through reinforcement learning. We evaluate our method on simple single- and two-object actions from the Something-Something dataset. Our approach allows an agent to learn actions from single videos, while watching multiple demonstrations makes the policy more robust. We show that policies learned in a simulated environment can be easily transferred to a real robot.

Related papers

• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)
• Learning Video-Conditioned Policies for Unseen Manipulation Tasks [83.2240629060453]
  Video-conditioned Policy learning maps human demonstrations of previously unseen tasks to robot manipulation skills. We learn our policy to generate appropriate actions given current scene observations and a video of the target task. We validate our approach on a set of challenging multi-task robot manipulation environments and outperform state of the art.
  arXiv  Detail & Related papers  (2023-05-10T16:25:42Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Learning Object Manipulation Skills from Video via Approximate Differentiable Physics [27.923004421974156]
  We teach robots to perform simple object manipulation tasks by watching a single video demonstration. A differentiable scene ensures perceptual fidelity between the 3D scene and the 2D video. We evaluate our approach on a 3D reconstruction task that consists of 54 video demonstrations.
  arXiv  Detail & Related papers  (2022-08-03T10:21:47Z)
• Learning Generalizable Dexterous Manipulation from Human Grasp Affordance [11.060931225148936]
  Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. Recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning. We propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category.
  arXiv  Detail & Related papers  (2022-04-05T16:26:22Z)
• A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation [63.1610540170754]
  We focus on the problem of visual non-prehensile planar manipulation. We propose a novel architecture that combines video decoding neural models with priors from contact mechanics. We find that our modular and fully differentiable architecture performs better than learning-only methods on unseen objects and motions.
  arXiv  Detail & Related papers  (2021-11-09T18:39:45Z)
• Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos [28.712673809577076]
  We present an approach for physical imitation from human videos for robot manipulation tasks. We design a perception module that learns to translate human videos to the robot domain followed by unsupervised keypoint detection. We evaluate the effectiveness of our approach on five robot manipulation tasks, including reaching, pushing, sliding, coffee making, and drawer closing.
  arXiv  Detail & Related papers  (2021-01-18T18:50:32Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.