DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation

• URL: http://arxiv.org/abs/2603.17851v1
• Date: Wed, 18 Mar 2026 15:39:58 GMT
• Title: DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
• Abstract summary: Large-scale, high-quality multimodal demonstrations are essential for robot learning of contact-rich dexterous manipulation.<n>We present DexViTac, a portable, human-centric data collection system tailored for contact-rich dexterous manipulation.
• Score: 4.463599051210204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale, high-quality multimodal demonstrations are essential for robot learning of contact-rich dexterous manipulation. While human-centric data collection systems lower the barrier to scaling, they struggle to capture the tactile information during physical interactions. Motivated by this, we present DexViTac, a portable, human-centric data collection system tailored for contact-rich dexterous manipulation. The system enables the high-fidelity acquisition of first-person vision, high-density tactile sensing, end-effector poses, and hand kinematics within unstructured, in-the-wild environments. Building upon this hardware, we propose a kinematics-grounded tactile representation learning algorithm that effectively resolves semantic ambiguities within tactile signals. Leveraging the efficiency of DexViTac, we construct a multimodal dataset comprising over 2,400 visuo-tactile-kinematic demonstrations. Experiments demonstrate that DexViTac achieves a collection efficiency exceeding 248 demonstrations per hour and remains robust against complex visual occlusions. Real-world deployment confirms that policies trained with the proposed dataset and learning strategy achieve an average success rate exceeding 85% across four challenging tasks. This performance significantly outperforms baseline methods, thereby validating the substantial improvement the system provides for learning contact-rich dexterous manipulation. Project page: https://xitong-c.github.io/DexViTac/.

Related papers

• STAR: Sparse Tactile Representation Learning in Vision-Tactile-Language-Action Models for Dexterous Manipulation [24.489844571043808]
  Dexterous manipulation requires coordinated multi-finger control and effective tactile feedback.<n>We build a robot platform and teleoperation system to collect a 200-hour bimanual dexterous manipulation dataset.<n>We propose STAR, an integrated training recipe for vision-tactile-language-action models.
  arXiv  Detail & Related papers  (2026-09-11T07:57:24Z)
• VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation [10.08235353271524]
  Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging.<n>This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale.<n>We introduce the VTOUCH dataset to provide high-fidelity physical interaction signals, adopt a matrix-style task design, and employ automated data collection pipelines covering real-world, demand-driven scenarios.
  arXiv  Detail & Related papers  (2026-04-22T11:08:08Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking [47.900334665380115]
  We propose UniVTAC, a simulation-based visuo-tactile data platform.<n>We introduce the UniVTAC encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data.<n>We present the UniVTAC Benchmark, which consists of eight representative visuo-tactile manipulation tasks.
  arXiv  Detail & Related papers  (2026-02-10T18:57:00Z)
• DexTac: Learning Contact-aware Visuotactile Policies via Hand-by-hand Teaching [14.892858615291717]
  We present DexTac, a visuo-tactile manipulation learning framework based on kinesthetic teaching.<n>We evaluate our framework on a challenging unimanual injection task.
  arXiv  Detail & Related papers  (2026-01-29T09:52:46Z)
• Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation [21.78866976181311]
  See-through-skin (STS) sensors combine tactile and visual perception.<n>Existing STS designs lack simultaneous multimodal perception and suffer from unreliable tactile tracking.<n>We introduce TacThru, an STS sensor enabling simultaneous visual perception and robust tactile signal extraction.
  arXiv  Detail & Related papers  (2025-12-10T17:35:13Z)
• DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation [25.208854363099352]
  This dataset contains 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations.<n>Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles.<n>Our real-to-sim pipeline uses reinforcement learning to train policies that control an actuated MANO hand in physics simulation.
  arXiv  Detail & Related papers  (2025-10-17T16:08:14Z)
• LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations [20.300415135664718]
  Long-horizon manipulation tasks require both physical dexterity and seamless sequencing of manipulation skills.<n>We propose a learning framework and system LodeStar that automatically decomposes task demonstrations into semantically meaningful skills.<n>Our approach significantly improves task performance and robustness compared to previous baselines.
  arXiv  Detail & Related papers  (2025-08-24T22:57:16Z)
• Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids [56.892520712892804]
  We introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three dexterous manipulation tasks.<n>We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors.
  arXiv  Detail & Related papers  (2025-02-27T18:59:52Z)
• Learning Visuotactile Skills with Two Multifingered Hands [80.99370364907278]
  We explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data.
  arXiv  Detail & Related papers  (2024-04-25T17:59:41Z)
• Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play [15.780086627089885]
  T-Dex is a new approach for tactile-based dexterity that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self-supervised tactile encoders. In the second phase, given a handful of demonstrations for a dexterous task, we learn non-parametric policies that combine the tactile observations with visual ones.
  arXiv  Detail & Related papers  (2023-03-21T17:59:20Z)
• Multi-dataset Training of Transformers for Robust Action Recognition [75.5695991766902]
  We study the task of robust feature representations, aiming to generalize well on multiple datasets for action recognition. Here, we propose a novel multi-dataset training paradigm, MultiTrain, with the design of two new loss terms, namely informative loss and projection loss. We verify the effectiveness of our method on five challenging datasets, Kinetics-400, Kinetics-700, Moments-in-Time, Activitynet and Something-something-v2.
  arXiv  Detail & Related papers  (2022-09-26T01:30:43Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.