AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning

• URL: http://arxiv.org/abs/2608.14028v1
• Date: Fri, 14 Aug 2026 07:19:06 GMT
• Title: AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
• Abstract summary: We present AdvDex, a framework for learning dexterous manipulation from human and robot demonstrations.<n>First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations.<n>Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an $mathrmSE(3)$ wrist pose and 15 finger joints.<n>Third, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation.
• Score: 25.241504281004243
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an $\mathrm{SE}(3)$ wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• HumanoidUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation [6.733928872257592]
  We propose HumanoidUMI, a portable and robot-free framework for humanoid whole-body data collection.<n>We use lightweight VR devices and UMI-inspired grippers to collect sparse human keypoint trajectories, wrist-view observations, and gripper actions.<n>Experiments in five real-world scenarios demonstrate the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2026-06-25T16:23:56Z)
• BridgeACT: Bridging Human Demonstrations to Robot Actions via Unified Tool-Target Affordances [11.402773793645244]
  We present BridgeACT, a framework that learns robotic manipulation directly from human videos without requiring robot demonstration data.<n>Our key idea is to model affordance as an embodiment-agnostic intermediate representation that bridges human demonstrations and robot actions.<n>The resulting affordances are mapped to robot actions through a grasping module and a lightweight closed-loop motion controller.
  arXiv  Detail & Related papers  (2026-04-25T11:01:27Z)
• ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration [51.69384671495837]
  ActiveGlasses is a system for learning robot manipulation from ego-centric human demonstrations with active vision.<n>A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference.<n>To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement.
  arXiv  Detail & Related papers  (2026-04-09T17:59:08Z)
• Cross-Hand Latent Representation for Vision-Language-Action Models [49.32460749933983]
  Training reliable vision-language-action models for dexterous manipulation requires large-scale demonstrations across many robotic hands.<n>We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands.
  arXiv  Detail & Related papers  (2026-03-10T18:50:57Z)
• DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos [56.64773686434068]
  We propose DexImit, an automated framework that converts human manipulation videos into physically plausible robot data.<n>DexImit can generate large-scale robot data based on human videos, either from the Internet or video generation models.<n>It is capable of handling diverse manipulation tasks, including tool use, long-horizon tasks, and fine-grained manipulations.
  arXiv  Detail & Related papers  (2026-02-10T18:59:02Z)
• METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model [36.82365894983052]
  A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills.<n>We propose METIS, a vision-language-action model for dexterous manipulation pretrained on egocentric datasets.<n>Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks.
  arXiv  Detail & Related papers  (2025-11-21T16:32:36Z)
• One-Shot Imitation under Mismatched Execution [7.060120660671016]
  Human demonstrations are a powerful way to program robots to do long-horizon manipulation tasks.<n> translating these demonstrations into robot-executable actions presents significant challenges due to execution mismatches in movement styles and physical capabilities.<n>We propose RHyME, a novel framework that automatically pairs human and robot trajectories using sequence-level optimal transport cost functions.
  arXiv  Detail & Related papers  (2024-09-10T16:11:57Z)
• RealDex: Towards Human-like Grasping for Robotic Dexterous Hand [64.33746404551343]
  We introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns.<n>RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios.
  arXiv  Detail & Related papers  (2024-02-21T14:59:46Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.