DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

• URL: http://arxiv.org/abs/2403.07788v2
• Date: Thu, 4 Jul 2024 04:35:04 GMT
• Title: DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
• Authors: Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C. Karen Liu,
• Abstract summary: Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. We introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data.
• Score: 28.37054959647664
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot hands. Beyond direct learning from human motion, DexCap also offers an optional human-in-the-loop correction mechanism during policy rollouts to refine and further improve task performance. Through extensive evaluation across six challenging dexterous manipulation tasks, our approach not only demonstrates superior performance but also showcases the system's capability to effectively learn from in-the-wild mocap data, paving the way for future data collection methods in the pursuit of human-level robot dexterity. More details can be found at https://dex-cap.github.io

Related papers

• EgoZero: Robot Learning from Smart Glasses [54.6168258133554]
  EgoZero learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses.<n>We deploy EgoZero policies on a Franka Panda robot and demonstrate zero-shot transfer with 70% success rate over 7 manipulation tasks.<n>Our results suggest that in-the-wild human data can serve as a scalable foundation for real-world robot learning.
  arXiv  Detail & Related papers  (2025-05-26T17:59:17Z)
• ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning [24.675197489823898]
  We introduce ManipTrans, a novel method for transferring human bimanual skills to dexterous robotic hands in simulation. Experiments show that ManipTrans surpasses state-of-the-art methods in success rate, fidelity, and efficiency. We also create DexManipNet, a large-scale dataset featuring previously unexplored tasks like pen capping and bottle unscrewing.
  arXiv  Detail & Related papers  (2025-03-27T17:50:30Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos. VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• DexterityGen: Foundation Controller for Unprecedented Dexterity [67.15251368211361]
  Teaching robots dexterous manipulation skills, such as tool use, presents a significant challenge. Current approaches can be broadly categorized into two strategies: human teleoperation (for imitation learning) and sim-to-real reinforcement learning. We introduce DexterityGen, which uses RL to pretrain large-scale dexterous motion primitives, such as in-hand rotation or translation. In the real world, we use human teleoperation as a prompt to the controller to produce highly dexterous behavior.
  arXiv  Detail & Related papers  (2025-02-06T18:49:35Z)
• DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning [42.88605563822155]
  We present a large-scale automated data generation system that synthesizes trajectories from human demonstrations for humanoid robots with dexterous hands. We generate 21K demos across these tasks from just 60 source human demos. We also present a real-to-sim-to-real pipeline and deploy it on a real-world humanoid can sorting task.
  arXiv  Detail & Related papers  (2024-10-31T17:48:45Z)
• RealDex: Towards Human-like Grasping for Robotic Dexterous Hand [64.47045863999061]
  We introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios.
  arXiv  Detail & Related papers  (2024-02-21T14:59:46Z)
• HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration [57.045140028275036]
  We show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning. We propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.
  arXiv  Detail & Related papers  (2022-12-08T15:56:13Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations [51.87067543670535]
  We propose a robot-learning system that can take a small number of human demonstrations and learn to grasp unseen object poses. We train a dexterous grasping policy that takes the point clouds of the object as input and predicts continuous actions to grasp objects from different initial robot states. The policy learned from our dataset can generalize well on unseen object poses in both simulation and the real world.
  arXiv  Detail & Related papers  (2022-09-28T17:51:49Z)
• MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control [15.848947335588301]
  We release MoCapAct, a dataset of expert agents and their rollouts, which contain proprioceptive observations and actions. We demonstrate the utility of MoCapAct by using it to train a single hierarchical policy capable of tracking the entire MoCap dataset within dm_control.
  arXiv  Detail & Related papers  (2022-08-15T17:57:33Z)
• From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation [26.738893736520364]
  We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. We construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-26T17:59:51Z)
• Model Predictive Control for Fluid Human-to-Robot Handovers [50.72520769938633]
  Planning motions that take human comfort into account is not a part of the human-robot handover process. We propose to generate smooth motions via an efficient model-predictive control framework. We conduct human-to-robot handover experiments on a diverse set of objects with several users.
  arXiv  Detail & Related papers  (2022-03-31T23:08:20Z)
• Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors [28.22210425264389]
  We investigate the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots. Our approach builds upon previous work on imitating human or dog Motion Capture (MoCap) data to learn a movement skill module.
  arXiv  Detail & Related papers  (2022-03-31T16:01:32Z)
• DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video [86.49357517864937]
  We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interaction videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose. We demonstrate that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment.
  arXiv  Detail & Related papers  (2022-02-01T00:45:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.