HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning

• URL: http://arxiv.org/abs/2606.04825v1
• Date: Wed, 03 Jun 2026 12:48:17 GMT
• Title: HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning
• Abstract summary: HapTile is a contact-grounded visuotactile manipulation dataset that advances beyond vision-only trajectory datasets.<n>The data collection platform integrates haptic feedback directly into the teleoperation controller, enabling the operator to perceive contact interactions in real time.<n>The dataset comprises everyday manipulation tasks spanning a broad range of contact-rich skills, including pick-and-place, folding, pressing, stacking, and other routine activities.
• Score: 24.64460457808404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the importance of tactile sensing for reliable manipulation, most existing Vision-Language-Action (VLA) datasets remain vision-only, and those that do incorporate tactile information typically lack the joint combination of task diversity, language conditioning, and action trajectories. Furthermore, existing teleoperation pipelines rarely provide haptic feedback to the operator, despite its established role in demonstration quality and manipulation stability. In this work, we present HapTile, a contact-grounded visuotactile manipulation dataset that advances beyond vision-only trajectory datasets by embedding physical interaction sensing at two levels: fingertip tactile feedback at the robot end-effector, and haptic-informed demonstrations at the teleoperator side. The data collection platform integrates haptic feedback directly into the teleoperation controller, enabling the operator to perceive contact interactions in real time. It is built around a standard and reproducible robotic system equipped with custom-designed fingertip tactile sensors. The dataset comprises everyday manipulation tasks spanning a broad range of contact-rich skills, including pick-and-place, folding, pressing, stacking, and other routine activities. Each task is paired with language instructions that condition the policy on the manipulation objective, together with synchronized visuotactile observations and action trajectories. In addition, we provide a benchmarking study on contact-rich policy learning using two baseline models to evaluate the effectiveness of the proposed contact-grounded dataset. The dataset and additional details are available on our website: haptile-dataset.github.io.

Related papers

• UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models [32.87879913150572]
  We propose a unified tactile learning framework for contact-rich manipulation.<n>We show that our method improves success rate, manipulation accuracy, and contact robustness over existing methods.
  arXiv  Detail & Related papers  (2026-06-30T14:24:00Z)
• Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms [70.51538670020267]
  We propose a hierarchical taxonomy that organizes the field into two primary dimensions: multimodal datasets and multimodal methods.<n>On the data side, we categorize resources ranging from Tactile-Vision datasets, Tactile-Language datasets, Tactile-Vision-Language datasets, and Tactile-Vision-Other datasets.<n>On the method side, we structure prior work into three core pillars: (1) Multimodal Perception and Recognition, (2) Cross-Modal Generation, focusing on bidirectional translation between tactile, vision, and text, and (3) Multimodal Interaction, emphasizing feedback control and language-guided manipulation.
  arXiv  Detail & Related papers  (2026-05-17T09:09:30Z)
• VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation [10.08235353271524]
  Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging.<n>This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale.<n>We introduce the VTOUCH dataset to provide high-fidelity physical interaction signals, adopt a matrix-style task design, and employ automated data collection pipelines covering real-world, demand-driven scenarios.
  arXiv  Detail & Related papers  (2026-04-22T11:08:08Z)
• TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks [16.262130849751312]
  TAMEn is a tactile-aware manipulation engine for closed-loop data collection in contact-rich tasks.<n>Our system features a cross-morphology wearable interface that enables rapid adaptation across heterogeneous grippers.<n>Our proposed visuo-tactile learning framework increases task success rates from 34% to 75% across diverse bimanual manipulation tasks.
  arXiv  Detail & Related papers  (2026-04-08T17:49:35Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper [7.618517580705364]
  We present a portable, lightweight gripper with integrated tactile sensors.<n>We propose a cross-modal representation learning framework that integrates visual and tactile signals.<n>We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer.
  arXiv  Detail & Related papers  (2025-07-20T17:53:59Z)
• Learning Visuotactile Skills with Two Multifingered Hands [80.99370364907278]
  We explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data.
  arXiv  Detail & Related papers  (2024-04-25T17:59:41Z)
• Touch and Go: Learning from Human-Collected Vision and Touch [16.139106833276]
  We propose a dataset with paired visual and tactile data called Touch and Go. Human data collectors probe objects in natural environments using tactile sensors. Our dataset spans a large number of "in the wild" objects and scenes.
  arXiv  Detail & Related papers  (2022-11-22T18:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.