DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation

• URL: http://arxiv.org/abs/2505.24853v1
• Date: Fri, 30 May 2025 17:50:23 GMT
• Title: DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation
• Authors: Zhao Mandi, Yifan Hou, Dieter Fox, Yashraj Narang, Ajay Mandlekar, Shuran Song,
• Abstract summary: We study the problem of functional discoterous manipulation policies to track object states from human hand-object demonstrations.<n>We propose a novel curriculum-based algorithm: the key idea is to use virtual object controllers with strength.<n>We release a simulation benchmark with a diverse set of tasks and dexterous hands, and show that DexMachina significantly outperforms baseline methods.
• Score: 48.68321200585559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the problem of functional retargeting: learning dexterous manipulation policies to track object states from human hand-object demonstrations. We focus on long-horizon, bimanual tasks with articulated objects, which is challenging due to large action space, spatiotemporal discontinuities, and embodiment gap between human and robot hands. We propose DexMachina, a novel curriculum-based algorithm: the key idea is to use virtual object controllers with decaying strength: an object is first driven automatically towards its target states, such that the policy can gradually learn to take over under motion and contact guidance. We release a simulation benchmark with a diverse set of tasks and dexterous hands, and show that DexMachina significantly outperforms baseline methods. Our algorithm and benchmark enable a functional comparison for hardware designs, and we present key findings informed by quantitative and qualitative results. With the recent surge in dexterous hand development, we hope this work will provide a useful platform for identifying desirable hardware capabilities and lower the barrier for contributing to future research. Videos and more at https://project-dexmachina.github.io/

Related papers

• MoDeSuite: Robot Learning Task Suite for Benchmarking Mobile Manipulation with Deformable Objects [17.54380759165508]
  We introduce MoDeSuite, the first task suite that addresses mobile manipulation tasks involving deformable objects.<n>Success in these tasks requires effective collaboration between the robot's base and manipulator, as well as the ability to exploit the deformability of the objects.<n>We train two state-of-the-art reinforcement learning algorithms and two imitation learning algorithms, highlighting the difficulties encountered and showing their performance in simulation.
  arXiv  Detail & Related papers  (2025-07-29T13:33:43Z)
• MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videos [43.836197294180316]
  We present MAPLE, a novel method for dexterous robotic manipulation that exploits rich manipulation priors to enable efficient policy learning.<n>Specifically, we predict hand-object contact points and detailed hand poses at the moment of hand-object contact and use the learned features to train policies for downstream manipulation tasks.
  arXiv  Detail & Related papers  (2025-04-08T14:25:25Z)
• CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World [20.52894595103719]
  CordViP is a novel framework that constructs and learns correspondences by leveraging the robust 6D pose estimation of objects and robot proprioception.<n>Our method demonstrates exceptional dexterous manipulation capabilities, achieving state-of-the-art performance in six real-world tasks.
  arXiv  Detail & Related papers  (2025-02-12T14:41:14Z)
• DexterityGen: Foundation Controller for Unprecedented Dexterity [67.15251368211361]
  Teaching robots dexterous manipulation skills, such as tool use, presents a significant challenge.<n>Current approaches can be broadly categorized into two strategies: human teleoperation (for imitation learning) and sim-to-real reinforcement learning.<n>We introduce DexterityGen, which uses RL to pretrain large-scale dexterous motion primitives, such as in-hand rotation or translation.<n>In the real world, we use human teleoperation as a prompt to the controller to produce highly dexterous behavior.
  arXiv  Detail & Related papers  (2025-02-06T18:49:35Z)
• GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation [12.940189262612677]
  GarmentLab is a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation.<n>Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators.<n>We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks.
  arXiv  Detail & Related papers  (2024-11-02T10:09:08Z)
• Interacting Hand-Object Pose Estimation via Dense Mutual Attention [97.26400229871888]
  3D hand-object pose estimation is the key to the success of many computer vision applications. We propose a novel dense mutual attention mechanism that is able to model fine-grained dependencies between the hand and the object. Our method is able to produce physically plausible poses with high quality and real-time inference speed.
  arXiv  Detail & Related papers  (2022-11-16T10:01:33Z)
• H-SAUR: Hypothesize, Simulate, Act, Update, and Repeat for Understanding Object Articulations from Interactions [62.510951695174604]
  "Hypothesize, Simulate, Act, Update, and Repeat" (H-SAUR) is a probabilistic generative framework that generates hypotheses about how objects articulate given input observations. We show that the proposed model significantly outperforms the current state-of-the-art articulated object manipulation framework. We further improve the test-time efficiency of H-SAUR by integrating a learned prior from learning-based vision models.
  arXiv  Detail & Related papers  (2022-10-22T18:39:33Z)
• Model Predictive Control for Fluid Human-to-Robot Handovers [50.72520769938633]
  Planning motions that take human comfort into account is not a part of the human-robot handover process. We propose to generate smooth motions via an efficient model-predictive control framework. We conduct human-to-robot handover experiments on a diverse set of objects with several users.
  arXiv  Detail & Related papers  (2022-03-31T23:08:20Z)
• Learning Dexterous Grasping with Object-Centric Visual Affordances [86.49357517864937]
  Dexterous robotic hands are appealing for their agility and human-like morphology. We introduce an approach for learning dexterous grasping. Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop.
  arXiv  Detail & Related papers  (2020-09-03T04:00:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.