V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos

• URL: http://arxiv.org/abs/2606.16436v1
• Date: Mon, 15 Jun 2026 09:08:11 GMT
• Title: V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos
• Abstract summary: V2P-Manip is a framework designed to learn dexterous manipulation policies directly from human demonstration videos.<n>We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning.
• Score: 31.75414004293948
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving autonomous robotic dexterous manipulation requires precise, human-like action sequences at scale. As a scalable supplement to costly teleoperation data, extracting trajectories with both visual fidelity and physical plausibility from monocular videos represents a promising frontier in embodied AI. To this end, we introduce V2P-Manip, an efficient framework designed to learn dexterous manipulation policies directly from human demonstration videos. We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning. To bridge the gap between visual perception and physical constraints, we introduce a two-stage refinement process to enforce spatial alignment and physical consistency. Evaluations on the TACO and OakInk benchmarks demonstrate that our approach significantly outperforms previous methods in pose accuracy, adaptability to unstructured environments, and training efficiency. Ultimately, experimental results confirm an average success rate of over 75% across multiple synthetic manipulation tasks and validate the adaptability of the extracted manipulation priors across diverse dexterous hand embodiments.

Related papers

• KAI: A Kinematic-Aware Interface for Data-Efficient Articulated Object Manipulation [63.408378267309786]
  Articulated object manipulation requires an understanding of kinematic structure that is difficult to learn from robot demonstrations alone.<n>We introduce the Kinematic-Aware Articulation Interface (KAI), a structured intermediate representation that captures the kinematic structure of articulated objects.<n>By embedding interpretable geometric and kinematic priors into policy learning, KAI provides a strong inductive bias aligned with the underlying structure of articulated motion.
  arXiv  Detail & Related papers  (2026-07-27T14:29:47Z)
• Contrastive Action-Image Pre-training for Visuomotor Control [91.5630572541332]
  We introduce CAIP, a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions.<n>Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders.
  arXiv  Detail & Related papers  (2026-06-15T20:00:20Z)
• From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data [71.22409934108924]
  Human videos are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation.<n>This survey provides a unified view of how human videos are transformed into effective knowledge for Vision-Language-Action (VLA) models.<n>We highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency.
  arXiv  Detail & Related papers  (2026-05-18T06:19:16Z)
• Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation [67.16145181790522]
  This paper tackles spatial perception and manipulation challenges in Vision-Language-Action (VLA) models.<n>We propose a Geometry-Guided Gated Transformer (G3T) that aligns multi-view features under 3D geometric guidance.<n>To improve action learning efficiency, we introduce Action Manifold Learning (AML), which directly predicts actions on the valid action manifold.
  arXiv  Detail & Related papers  (2026-05-12T09:21:29Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild [37.43820830488286]
  We present JALA, a pretraining framework that learns Jointly-Aligned Latent Actions.<n>We scale this approach with UniHand-Mix, a 7.5M video corpus (>2,000 hours) blending laboratory and in-the-wild footage.<n>Experiments demonstrate that JALA generates more realistic hand motions in both controlled and unconstrained scenarios.
  arXiv  Detail & Related papers  (2026-02-25T09:46:42Z)
• ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation [27.54751123419347]
  We propose ConLA, an unsupervised pretraining framework for learning robotic policies from human videos.<n>By pretraining solely on human videos, our method for the first time surpasses the performance obtained with real robot trajectory pretraining.
  arXiv  Detail & Related papers  (2026-01-31T06:40:57Z)
• METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model [36.82365894983052]
  A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills.<n>We propose METIS, a vision-language-action model for dexterous manipulation pretrained on egocentric datasets.<n>Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks.
  arXiv  Detail & Related papers  (2025-11-21T16:32:36Z)
• Precise Action-to-Video Generation Through Visual Action Prompts [62.951609704196485]
  Action-driven video generation faces a precision-generality trade-off.<n>Agent-centric action signals provide precision at the cost of cross-domain transferability.<n>We "render" actions into precise visual prompts as domain-agnostic representations.
  arXiv  Detail & Related papers  (2025-08-18T17:12:28Z)
• Physical Autoregressive Model for Robotic Manipulation without Action Pretraining [65.8971623698511]
  We build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR)<n>PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining.<n>Experiments on the ManiSkill benchmark show that PAR achieves a 100% success rate on the PushCube task.
  arXiv  Detail & Related papers  (2025-08-13T13:54:51Z)
• Reconstructing Close Human Interaction with Appearance and Proxemics Reasoning [50.76723760768117]
  Existing human pose estimation methods cannot recover plausible close interactions from in-the-wild videos.<n>We find that human appearance can provide a straightforward cue to address these obstacles.<n>We propose a dual-branch optimization framework to reconstruct accurate interactive motions with plausible body contacts constrained by human appearances, social proxemics, and physical laws.
  arXiv  Detail & Related papers  (2025-07-03T12:19:26Z)
• VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation [8.882764358932276]
  Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination.<n>Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills.<n>We introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation.
  arXiv  Detail & Related papers  (2025-01-07T08:14:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.