Cross-Embodiment Dexterous Grasping with Reinforcement Learning

• URL: http://arxiv.org/abs/2410.02479v1
• Date: Thu, 3 Oct 2024 13:36:02 GMT
• Title: Cross-Embodiment Dexterous Grasping with Reinforcement Learning
• Abstract summary: Dexterous hands exhibit significant potential for complex real-world grasping tasks. We study the learning of cross-embodiment dexterous grasping policies using reinforcement learning. Our approach demonstrates an 80% success rate in grasping objects from the YCB dataset.
• Score: 29.867524903691802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous hands exhibit significant potential for complex real-world grasping tasks. While recent studies have primarily focused on learning policies for specific robotic hands, the development of a universal policy that controls diverse dexterous hands remains largely unexplored. In this work, we study the learning of cross-embodiment dexterous grasping policies using reinforcement learning (RL). Inspired by the capability of human hands to control various dexterous hands through teleoperation, we propose a universal action space based on the human hand's eigengrasps. The policy outputs eigengrasp actions that are then converted into specific joint actions for each robot hand through a retargeting mapping. We simplify the robot hand's proprioception to include only the positions of fingertips and the palm, offering a unified observation space across different robot hands. Our approach demonstrates an 80% success rate in grasping objects from the YCB dataset across four distinct embodiments using a single vision-based policy. Additionally, our policy exhibits zero-shot generalization to two previously unseen embodiments and significant improvement in efficient finetuning. For further details and videos, visit our project page https://sites.google.com/view/crossdex.

Related papers

• AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning [25.241504281004243]
  We present AdvDex, a framework for learning dexterous manipulation from human and robot demonstrations.<n>First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations.<n>Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an $mathrmSE(3)$ wrist pose and 15 finger joints.<n>Third, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation.
  arXiv  Detail & Related papers  (2026-08-14T07:19:06Z)
• Towards Human-level Dexterous Teleoperation [62.77371751776912]
  Human-level dexterous teleoperation requires precise control of object motion via dynamic hand-object contact.<n>We introduce TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution.<n>We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands.
  arXiv  Detail & Related papers  (2026-07-13T12:36:47Z)
• AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance [52.973947477815614]
  AnyDexRT is a calibration-free method for intuitive dexterous teleoperation across human-like dexterous hands.<n>Experiments show that AnyDexRT improves quality, reduces manual tuning, and provides more intuitive and efficient control than prior methods.
  arXiv  Detail & Related papers  (2026-07-09T10:29:53Z)
• Cross-Embodiment Robot Manipulation via a Unified Hand Action Space [5.6990246782356735]
  We propose a sphere-based unified action representation for cross-embodiment dexterous manipulation.<n>Our experiments show that the proposed UHAS representation enables stable dexterous control and cross-embodiment policy transfer across robotic hands.
  arXiv  Detail & Related papers  (2026-07-03T19:19:20Z)
• Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations [47.549615826832856]
  Dexterous Point Policy learns dexterous manipulation policies directly from human videos and requires no robot demonstrations.<n>We extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints.<n>On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%.
  arXiv  Detail & Related papers  (2026-06-09T09:13:36Z)
• DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations [30.95930862562584]
  We propose DexRep, a novel hand-object interaction representation to capture object surface features and spatial relations between hands and objects.<n>Based on DexRep, policies are learned for three dexterous manipulation tasks, i.e. grasping, in-hand reorientation, bimanual handover, and experiments are conducted.<n>For grasping, the policy learned with 40 objects achieves a success rate of 87.9% on more than 5000 unseen objects of diverse categories.<n>For the in-hand reorientation and handover tasks, the policies also boost the success rates and other metrics of existing hand-object representations by 20% to 40%
  arXiv  Detail & Related papers  (2026-02-25T11:38:07Z)
• Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction [24.49384094440561]
  We propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos.<n>In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand.<n>In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand.
  arXiv  Detail & Related papers  (2026-02-09T18:56:02Z)
• Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations [52.29884993824894]
  Learning multi-fingered robot policies from humans performing daily tasks in natural environments has long been a grand goal in the robotics community.<n>AINA enables learning multi-fingered policies from data collected by anyone, anywhere, and in any environment using Aria Gen 2 glasses.
  arXiv  Detail & Related papers  (2025-11-20T18:59:02Z)
• AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiency [49.868970174484204]
  We introduce an efficient approach for learning dexterous grasping with minimal data. Our method achieves high performance with human-level learning efficiency: only hundreds of grasp attempts on 40 training objects. This method demonstrates promising applications for humanoid robots, prosthetics, and other domains requiring robust, versatile robotic manipulation.
  arXiv  Detail & Related papers  (2025-02-23T03:26:06Z)
• FunGrasp: Functional Grasping for Diverse Dexterous Hands [8.316017819784603]
  We introduce FunGrasp, a system that enables functional dexterous grasping across various robot hands. To achieve robust sim-to-real transfer, we employ several techniques including privileged learning, system identification, domain randomization, and gravity compensation.
  arXiv  Detail & Related papers  (2024-11-24T07:30:54Z)
• OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation [35.97702591413093]
  We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately.
  arXiv  Detail & Related papers  (2024-10-15T17:17:54Z)
• Hand-Object Interaction Pretraining from Videos [77.92637809322231]
  We learn general robot manipulation priors from 3D hand-object interaction trajectories. We do so by sharing both the human hand and the manipulated object in 3D space and human motions to robot actions. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches.
  arXiv  Detail & Related papers  (2024-09-12T17:59:07Z)
• Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation [49.03165169369552]
  By training a single policy across many different kinds of robots, a robot learning method can leverage much broader and more diverse datasets. We propose CrossFormer, a scalable and flexible transformer-based policy that can consume data from any embodiment. We demonstrate that the same network weights can control vastly different robots, including single and dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds.
  arXiv  Detail & Related papers  (2024-08-21T17:57:51Z)
• Dreamitate: Real-World Visuomotor Policy Learning via Video Generation [49.03287909942888]
  We propose a visuomotor policy learning framework that fine-tunes a video diffusion model on human demonstrations of a given task. We generate an example of an execution of the task conditioned on images of a novel scene, and use this synthesized execution directly to control the robot.
  arXiv  Detail & Related papers  (2024-06-24T17:59:45Z)
• Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans [58.27029676638521]
  We show how passive human videos can serve as a rich source of data for learning such generalist robots. We learn a human plan predictor that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects.
  arXiv  Detail & Related papers  (2023-12-01T18:54:12Z)
• Learning a Universal Human Prior for Dexterous Manipulation from Human Preference [35.54663426598218]
  We propose a framework that learns a universal human prior using direct human preference feedback over videos. A task-agnostic reward model is trained through iteratively generating diverse polices and collecting human preference over the trajectories. Our method empirically demonstrates more human-like behaviors on robot hands in diverse tasks including even unseen tasks.
  arXiv  Detail & Related papers  (2023-04-10T14:17:33Z)
• From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation [26.738893736520364]
  We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. We construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-26T17:59:51Z)
• DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video [86.49357517864937]
  We propose DexVIP, an approach to learn dexterous robotic grasping from human-object interaction videos. We do this by curating grasp images from human-object interaction videos and imposing a prior over the agent's hand pose. We demonstrate that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment.
  arXiv  Detail & Related papers  (2022-02-01T00:45:57Z)
• Learning Dexterous Grasping with Object-Centric Visual Affordances [86.49357517864937]
  Dexterous robotic hands are appealing for their agility and human-like morphology. We introduce an approach for learning dexterous grasping. Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop.
  arXiv  Detail & Related papers  (2020-09-03T04:00:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.