Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

• URL: http://arxiv.org/abs/2606.29941v1
• Date: Mon, 29 Jun 2026 08:20:18 GMT
• Title: Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation
• Abstract summary: Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation.<n>Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge.<n>We propose a motion-aware tactile representation to facilitate contact-rich manipulation.
• Score: 74.87716678195099
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.

Related papers

• AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion [60.79507611737292]
  Humans achieve stable and adaptive grasps by seamlessly integrating visual perception and tactile feedback.<n>Existing robotic grasping approaches rely on visual inputs and lack mechanisms for tactile-guided adaptation after contact.<n>We propose a unified visuo-tactile-fusion grasping framework that integrates grasp generation, feasibility prediction, and adaptive refinement.
  arXiv  Detail & Related papers  (2026-08-06T16:29:11Z)
• UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models [32.87879913150572]
  We propose a unified tactile learning framework for contact-rich manipulation.<n>We show that our method improves success rate, manipulation accuracy, and contact robustness over existing methods.
  arXiv  Detail & Related papers  (2026-06-30T14:24:00Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• Multi-Modal Manipulation via Multi-Modal Policy Consensus [62.49978559936122]
  We propose a new approach to integrate diverse sensory modalities for robotic manipulation.<n>Our method factorizes the policy into a set of diffusion models, each specialized for a single representation.<n>We evaluate our approach on simulated manipulation tasks in RLBench, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion.
  arXiv  Detail & Related papers  (2025-09-27T19:43:04Z)
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper [7.618517580705364]
  We present a portable, lightweight gripper with integrated tactile sensors.<n>We propose a cross-modal representation learning framework that integrates visual and tactile signals.<n>We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer.
  arXiv  Detail & Related papers  (2025-07-20T17:53:59Z)
• AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch [9.606323817785114]
  We present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. Rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy.
  arXiv  Detail & Related papers  (2024-05-12T22:51:35Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.