UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data

• URL: http://arxiv.org/abs/2609.16504v1
• Date: Tue, 15 Sep 2026 01:52:33 GMT
• Title: UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data
• Abstract summary: We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations.<n>We collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist.<n>The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline.
• Score: 0.45880283710344055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human-object interaction references to a robotic arm-hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions. Project page: https://unidex-vitac.github.io/

Related papers

• Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions [6.968853416384178]
  Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion.<n>We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations.
  arXiv  Detail & Related papers  (2026-09-15T06:04:36Z)
• Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction [30.46515328565822]
  We present DEX-X, a framework for learning visual-tactile dexterous manipulation from human videos through simulation.<n>We reconstruct hand-object interactions in simulation, where physically grounded contact dynamics provide tactile supervision unavailable in the original videos.<n>We demonstrate zero-shot sim-to-real transfer on a dexterous hand-arm platform across diverse grasping and contact-rich tool-use tasks.
  arXiv  Detail & Related papers  (2026-09-07T16:47:39Z)
• Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation [0.26763498831034044]
  This paper presents a framework for learning human-like robot motion from demonstration.<n>A dataset of 3,142 handwriting demonstrations was collected from 22 participants.<n>A user study with 21 participants evaluated the perceived human-likeness of the generated trajectories.
  arXiv  Detail & Related papers  (2026-08-06T16:12:18Z)
• Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations [47.549615826832856]
  Dexterous Point Policy learns dexterous manipulation policies directly from human videos and requires no robot demonstrations.<n>We extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints.<n>On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%.
  arXiv  Detail & Related papers  (2026-06-09T09:13:36Z)
• Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation [5.967530183571141]
  Human-to-Robot'' imitation learning pipeline enables robots to acquire manipulation skills directly from unstructured video demonstrations.<n>Key innovation is a modular framework that decouples the learning process into two distinct stages.<n>For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations.
  arXiv  Detail & Related papers  (2026-02-22T13:26:27Z)
• DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos [56.64773686434068]
  We propose DexImit, an automated framework that converts human manipulation videos into physically plausible robot data.<n>DexImit can generate large-scale robot data based on human videos, either from the Internet or video generation models.<n>It is capable of handling diverse manipulation tasks, including tool use, long-horizon tasks, and fine-grained manipulations.
  arXiv  Detail & Related papers  (2026-02-10T18:59:02Z)
• Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction [24.49384094440561]
  We propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos.<n>In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand.<n>In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand.
  arXiv  Detail & Related papers  (2026-02-09T18:56:02Z)
• HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos [74.43500240121476]
  We present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids.<n>HumanX achieves over 8 times higher generalization success than prior methods.
  arXiv  Detail & Related papers  (2026-02-02T18:53:01Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• Feel the Force: Contact-Driven Learning from Humans [52.36160086934298]
  Controlling fine-grained forces during manipulation remains a core challenge in robotics.<n>We present FeelTheForce, a robot learning system that models human tactile behavior to learn force-sensitive manipulation.<n>Our approach grounds robust low-level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks.
  arXiv  Detail & Related papers  (2025-06-02T17:57:52Z)
• Toward Aligning Human and Robot Actions via Multi-Modal Demonstration Learning [3.9738951919572827]
  We propose a framework that explicitly models human demonstrations from RGB video with robot demonstrations in voxelized RGB-D space.<n>Our approach combines ResNet-based visual encoding for human intention modeling and a Perceiver Transformer for voxel-based robot action prediction.
  arXiv  Detail & Related papers  (2025-04-14T21:14:51Z)
• Learning Manipulation by Predicting Interaction [85.57297574510507]
  We propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction. The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms.
  arXiv  Detail & Related papers  (2024-06-01T13:28:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.