Towards Human-level Dexterous Teleoperation

• URL: http://arxiv.org/abs/2607.11481v1
• Date: Mon, 13 Jul 2026 12:36:47 GMT
• Title: Towards Human-level Dexterous Teleoperation
• Abstract summary: Human-level dexterous teleoperation requires precise control of object motion via dynamic hand-object contact.<n>We introduce TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution.<n>We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands.
• Score: 62.77371751776912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.

Related papers

• AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance [52.973947477815614]
  AnyDexRT is a calibration-free method for intuitive dexterous teleoperation across human-like dexterous hands.<n>Experiments show that AnyDexRT improves quality, reduces manual tuning, and provides more intuitive and efficient control than prior methods.
  arXiv  Detail & Related papers  (2026-07-09T10:29:53Z)
• Immersive Social Interaction with VR and LLM-Assisted Humanoids [8.355232835265982]
  This paper presents an immersive teleoperation framework that integrates voice-controlled locomotion, VR-based manipulation, and bidirectional social interaction for whole-body humanoid control.<n>Using Apple Vision Pro, the operator receives egocentric visual feedback, issues natural-language locomotion commands, and teleoperates the robot's arms and dexterous hands through wrist and finger tracking.<n>We evaluate the framework on a Unitree H1 humanoid equipped with dexterous hands in manipulation and social interaction tasks.
  arXiv  Detail & Related papers  (2026-07-08T14:00:51Z)
• DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy [6.652421441096025]
  Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics.<n>Traditional teleoperation systems fail in contact-rich tasks because embodiment gaps hinder accurate kinematic mapping.<n>We propose a tactile-driven adaptation strategy designed to enable fine-grained manipulation on top of teleoperation pipelines.
  arXiv  Detail & Related papers  (2026-06-22T14:48:58Z)
• RealDexUMI: A Wearable Universal Manipulation Interface for Dexterous Robot Learning [21.875676616617838]
  We present RealDexUMI, a wearable universal manipulation interface built around a shared dexterous end-effector module.<n>A palm-side is teleoperation maps human finger inputs to robot-hand commands, enabling real-time, glove-free, intuitive, and precise hand control.
  arXiv  Detail & Related papers  (2026-06-04T11:28:23Z)
• UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos [65.2981273885678]
  Dexterous manipulation remains challenging due to the cost of collecting real-robot teleoperation data.<n>We present UniDex, a robot foundation suite that couples a large-scale robot-centric dataset with a unified vision-language-action (VLA) policy.<n>UniDex-Dataset, UniDex-VLA, and UniDex-Cap provide a scalable foundation suite for universal dexterous manipulation.
  arXiv  Detail & Related papers  (2026-03-23T17:49:12Z)
• CLOT: Closed-Loop Global Motion Tracking for Whole-Body Humanoid Teleoperation [54.7399209456857]
  We present CLOT, a real-time whole-body humanoid teleoperation system that achieves closed-loop global motion tracking.<n>CLOT synchronizes operator and robot poses in a closed loop, enabling drift-free human-to-humanoid mimicry over long timehorizons.<n>We propose a data-driven randomization strategy that decouples observation trajectories from reward evaluation, enabling smooth and stable global corrections.
  arXiv  Detail & Related papers  (2026-02-13T12:03:13Z)
• ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning [24.675197489823898]
  We introduce ManipTrans, a novel method for transferring human bimanual skills to dexterous robotic hands in simulation.<n>Experiments show that ManipTrans surpasses state-of-the-art methods in success rate, fidelity, and efficiency.<n>We also create DexManipNet, a large-scale dataset featuring previously unexplored tasks like pen capping and bottle unscrewing.
  arXiv  Detail & Related papers  (2025-03-27T17:50:30Z)
• DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation [78.60543357822957]
  Dexterous manipulation with contact-rich interactions is crucial for advanced robotics.<n>We introduce DexHandDiff, an interaction-aware diffusion planning framework for adaptive dexterous manipulation.<n>Our framework achieves an average of 70.7% success rate on goal adaptive dexterous tasks, highlighting its robustness and flexibility in contact-rich manipulation.
  arXiv  Detail & Related papers  (2024-11-27T18:03:26Z)
• RealDex: Towards Human-like Grasping for Robotic Dexterous Hand [64.33746404551343]
  We introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns.<n>RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios.
  arXiv  Detail & Related papers  (2024-02-21T14:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.