FlowTouch: View-Invariant Visuo-Tactile Prediction

• URL: http://arxiv.org/abs/2603.08255v1
• Date: Mon, 09 Mar 2026 11:24:29 GMT
• Title: FlowTouch: View-Invariant Visuo-Tactile Prediction
• Abstract summary: In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction.<n>Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns.<n>Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances.
• Score: 35.30526041510819
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/

Related papers

• Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation [74.87716678195099]
  Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation.<n>Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge.<n>We propose a motion-aware tactile representation to facilitate contact-rich manipulation.
  arXiv  Detail & Related papers  (2026-06-29T08:20:18Z)
• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data [10.059624183053499]
  Visuo-tactile perception aims to understand an object's tactile properties, such as texture, softness, and rigidity.<n>We introduce RA-Touch, a retrieval-augmented framework that improves visuo-tactile perception by leveraging visual data enriched with tactile semantics.
  arXiv  Detail & Related papers  (2025-05-20T12:23:21Z)
• PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation [8.997347199266592]
  We introduce PseudoTouch which links high-dimensional structural information to low-dimensional sensor signals.<n>It does so by learning a low-dimensional visual-tactile embedding, wherein we encode a depth patch from which we decode the tactile signal.<n>We demonstrate the utility of our trained PseudoTouch model in two downstream tasks: object recognition and grasp stability prediction.
  arXiv  Detail & Related papers  (2024-03-22T10:51:31Z)
• Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control [12.302685367517718]
  High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. We study a new concept: textittactile saliency for robot touch, inspired by the human touch attention mechanism from neuroscience.
  arXiv  Detail & Related papers  (2023-07-26T21:19:45Z)
• Controllable Visual-Tactile Synthesis [28.03469909285511]
  We develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We then introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device.
  arXiv  Detail & Related papers  (2023-05-04T17:59:51Z)
• TactoFind: A Tactile Only System for Object Retrieval [14.732140705441992]
  We study the problem of object retrieval in scenarios where visual sensing is absent. Unlike vision, where cameras can observe the entire scene, touch sensors are local and only observe parts of the scene that are in contact with the manipulator. We present a system capable of using sparse tactile feedback from fingertip touch sensors on a dexterous hand to localize, identify and grasp novel objects without any visual feedback.
  arXiv  Detail & Related papers  (2023-03-23T17:50:09Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• OmniTact: A Multi-Directional High Resolution Touch Sensor [109.28703530853542]
  Existing tactile sensors are either flat, have small sensitive fields or only provide low-resolution signals. We introduce OmniTact, a multi-directional high-resolution tactile sensor. We evaluate the capabilities of OmniTact on a challenging robotic control task.
  arXiv  Detail & Related papers  (2020-03-16T01:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.