NestDex: Nested Policy Learning with Copilot Assisted Teleoperation for Dexterous Manipulation

• URL: http://arxiv.org/abs/2608.13362v1
• Date: Thu, 13 Aug 2026 15:25:39 GMT
• Title: NestDex: Nested Policy Learning with Copilot Assisted Teleoperation for Dexterous Manipulation
• Abstract summary: Dexterous manipulation promises substantially richer robot interaction with the physical world.<n>But learning these behaviours remains constrained by the difficulty of collecting consistent, complete-task demonstrations.<n>We introduce NestDex, a nested policy-learning framework that reduces this burden by using learned hand skills to assist demonstration collection.
• Score: 3.901334865327756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dexterous manipulation promises substantially richer robot interaction with the physical world, but learning these behaviours remains constrained by the difficulty of collecting consistent, complete-task demonstrations. Unlike parallel-jaw manipulation, dexterous tasks require the operator to coordinate arm motion with precise, contact-rich finger behaviour throughout the task. We introduce NestDex, a nested policy-learning framework that reduces this burden by using learned hand skills to assist demonstration collection. The operator controls the arm and regulates the active hand skill through a single-DoF clutch, rather than directly specifying the full finger trajectory. The inner hand policy adapts its motion from the latest proprioceptive history, while a vision-language selector activates the appropriate skill for each task stage. The resulting demonstrations train a separate outer visuomotor policy that controls both the arm and hand without the inner policies at deployment. A hand-action variational autoencoder provides compact hand-action targets while retaining arm commands in joint space. Across real-world dexterous manipulation experiments, NestDex improves demonstration reliability and efficiency, and the resulting empirical evaluations support effective autonomous policy learning. Video Demo are available at project website https://aus.bot/research/nestdex.

Related papers

• AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning [25.241504281004243]
  We present AdvDex, a framework for learning dexterous manipulation from human and robot demonstrations.<n>First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations.<n>Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an $mathrmSE(3)$ wrist pose and 15 finger joints.<n>Third, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation.
  arXiv  Detail & Related papers  (2026-08-14T07:19:06Z)
• Towards Human-level Dexterous Teleoperation [62.77371751776912]
  Human-level dexterous teleoperation requires precise control of object motion via dynamic hand-object contact.<n>We introduce TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution.<n>We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands.
  arXiv  Detail & Related papers  (2026-07-13T12:36:47Z)
• AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance [52.973947477815614]
  AnyDexRT is a calibration-free method for intuitive dexterous teleoperation across human-like dexterous hands.<n>Experiments show that AnyDexRT improves quality, reduces manual tuning, and provides more intuitive and efficient control than prior methods.
  arXiv  Detail & Related papers  (2026-07-09T10:29:53Z)
• RealDexUMI: A Wearable Universal Manipulation Interface for Dexterous Robot Learning [21.875676616617838]
  We present RealDexUMI, a wearable universal manipulation interface built around a shared dexterous end-effector module.<n>A palm-side is teleoperation maps human finger inputs to robot-hand commands, enabling real-time, glove-free, intuitive, and precise hand control.
  arXiv  Detail & Related papers  (2026-06-04T11:28:23Z)
• Coordinated Humanoid Manipulation with Choice Policies [43.635012621027904]
  We present a system that combines a modular teleoperation interface with a scalable learning framework.<n>Choice Policy is an imitation learning approach that generates multiple candidate actions and learns to score them.<n>We validate our approach on two real-world tasks: loading dishwasher and whole-body loco-manipulation for whiteboard wiping.
  arXiv  Detail & Related papers  (2025-12-31T18:59:53Z)
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration [58.4036440289082]
  Hand-object motion-capture (MoCap) offer large-scale, contact-rich demonstrations and hold promise for dexterous robotic scopes.<n>We introduce Dexplore, a unified single-loop optimization that performs repositories and tracking to learn robot control policies directly from MoCap at scale.
  arXiv  Detail & Related papers  (2025-09-11T17:59:07Z)
• HannesImitation: Grasping with the Hannes Prosthetic Hand via Imitation Learning [5.122722600158078]
  In robotics, imitation learning has emerged as a promising approach for learning grasping and complex manipulation tasks.<n>We present HannesImitationPolicy, an imitation learning-based method to control the Hannes prosthetic hand.<n>We leverage such data to train a single diffusion policy and deploy it on the prosthetic hand to predict the wrist orientation and hand closure for grasping.
  arXiv  Detail & Related papers  (2025-08-01T10:09:38Z)
• Feel the Force: Contact-Driven Learning from Humans [52.36160086934298]
  Controlling fine-grained forces during manipulation remains a core challenge in robotics.<n>We present FeelTheForce, a robot learning system that models human tactile behavior to learn force-sensitive manipulation.<n>Our approach grounds robust low-level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks.
  arXiv  Detail & Related papers  (2025-06-02T17:57:52Z)
• Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control [67.23580984118479]
  Imitation Learning (IL) is an effective framework to learn visuomotor skills from offline demonstration data. Hand-eye Action Networks (HAN) can approximate human's hand-eye coordination behaviors by learning from human teleoperated demonstrations.
  arXiv  Detail & Related papers  (2021-02-28T01:49:13Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.