A Visuo-Tactile Data Collection System with Haptic Feedback for Coarse-to-Fine Imitation Learning

• URL: http://arxiv.org/abs/2605.08757v1
• Date: Sat, 09 May 2026 07:38:38 GMT
• Title: A Visuo-Tactile Data Collection System with Haptic Feedback for Coarse-to-Fine Imitation Learning
• Abstract summary: We present a visuo-tactile data-collection system that generates temporally structured, contact-rich demonstrations for imitation learning.<n>Our system introduces a direct-drive gripper that the operator actuates with the fingers, preserving natural haptic feedback.
• Score: 5.960094442546868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a visuo-tactile data-collection system that generates temporally structured, contact-rich demonstrations for imitation learning. Conventional systems often decouple the operator from contact forces, which hinders the demonstration of subtle force modulation. Our system introduces a direct-drive gripper that the operator actuates with the fingers, preserving natural haptic feedback. Integrated visual sensors and custom tactile arrays capture image streams and contact geometry. A handle-mounted push button enables the operator to annotate the task's temporal structure in real time by marking task-critical regions. By fusing in-hand force perception with in-situ temporal annotation, the system produces multimodal datasets designed for coarse-to-fine learning algorithms that exploit structural task knowledge, enabling the development of high-quality manipulation policies.

Related papers

• AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion [60.79507611737292]
  Humans achieve stable and adaptive grasps by seamlessly integrating visual perception and tactile feedback.<n>Existing robotic grasping approaches rely on visual inputs and lack mechanisms for tactile-guided adaptation after contact.<n>We propose a unified visuo-tactile-fusion grasping framework that integrates grasp generation, feasibility prediction, and adaptive refinement.
  arXiv  Detail & Related papers  (2026-08-06T16:29:11Z)
• τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision [43.426795127841075]
  We present, a touch-augmented Vision-Language-Action (VLA) framework that learns an action-conditioned tactile representation from future visual supervision.<n>We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-07-27T14:25:15Z)
• HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning [24.64460457808404]
  HapTile is a contact-grounded visuotactile manipulation dataset that advances beyond vision-only trajectory datasets.<n>The data collection platform integrates haptic feedback directly into the teleoperation controller, enabling the operator to perceive contact interactions in real time.<n>The dataset comprises everyday manipulation tasks spanning a broad range of contact-rich skills, including pick-and-place, folding, pressing, stacking, and other routine activities.
  arXiv  Detail & Related papers  (2026-06-03T12:48:17Z)
• TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks [16.262130849751312]
  TAMEn is a tactile-aware manipulation engine for closed-loop data collection in contact-rich tasks.<n>Our system features a cross-morphology wearable interface that enables rapid adaptation across heterogeneous grippers.<n>Our proposed visuo-tactile learning framework increases task success rates from 34% to 75% across diverse bimanual manipulation tasks.
  arXiv  Detail & Related papers  (2026-04-08T17:49:35Z)
• Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity [1.6865201067073452]
  We introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems.<n>Koopman-UBM is a first instantiation of UBMs that learns a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system.
  arXiv  Detail & Related papers  (2026-02-07T07:18:00Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• ImplicitRDP: An End-to-End Visual-Force Diffusion Policy with Structural Slow-Fast Learning [52.86018040861575]
  We propose a unified end-to-end visual-force diffusion policy that integrates visual planning and reactive force control within a single network.<n>We introduce Structural Slow-Fast Learning, a mechanism utilizing causal attention to simultaneously process asynchronous visual and force tokens.<n>Experiments on contact-rich tasks demonstrate that ImplicitRDP significantly outperforms both vision-only and hierarchical baselines.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• Reactive In-Air Clothing Manipulation with Confidence-Aware Dense Correspondence and Visuotactile Affordance [2.9070495263473375]
  Prior systems often flatten garments or assume visibility of key features.<n>We present a dual-arm visuotactile framework that combines confidence-aware dense visual correspondence and tactile-supervised grasp affordance.<n>We demonstrate our task-agnostic grasp selection module in folding and hanging tasks.
  arXiv  Detail & Related papers  (2025-09-04T05:16:56Z)
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper [7.618517580705364]
  We present a portable, lightweight gripper with integrated tactile sensors.<n>We propose a cross-modal representation learning framework that integrates visual and tactile signals.<n>We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer.
  arXiv  Detail & Related papers  (2025-07-20T17:53:59Z)
• Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation [58.95799126311524]
  Humans can accomplish contact-rich tasks using vision and touch, with highly reactive capabilities such as fast response to external changes and adaptive control of contact forces.<n>Existing visual imitation learning approaches rely on action chunking to model complex behaviors.<n>We introduce TactAR, a low-cost teleoperation system that provides real-time tactile feedback through Augmented Reality.
  arXiv  Detail & Related papers  (2025-03-04T18:58:21Z)
• Temporal Binding Foundation Model for Material Property Recognition via Tactile Sequence Perception [2.3724852180691025]
  This letter presents a novel approach leveraging a temporal binding foundation model for tactile sequence understanding.<n>The proposed system captures the sequential nature of tactile interactions, similar to human fingertip perception.
  arXiv  Detail & Related papers  (2025-01-24T21:47:38Z)
• Robust Collaborative Perception without External Localization and Clock Devices [52.32342059286222]
  A consistent spatial-temporal coordination across multiple agents is fundamental for collaborative perception. Traditional methods depend on external devices to provide localization and clock signals. We propose a novel approach: aligning by recognizing the inherent geometric patterns within the perceptual data of various agents.
  arXiv  Detail & Related papers  (2024-05-05T15:20:36Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.