Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation

• URL: http://arxiv.org/abs/2602.13833v1
• Date: Sat, 14 Feb 2026 16:05:08 GMT
• Title: Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation
• Abstract summary: Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
• Score: 82.63833405368159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generalizing tool manipulation requires both semantic planning and precise physical control. Modern generalist robot policies, such as Vision-Language-Action (VLA) models, often lack the high-fidelity physical grounding required for contact-rich tool manipulation. Conversely, existing contact-aware policies that leverage tactile or haptic sensing are typically instance-specific and fail to generalize across diverse tool geometries. Bridging this gap requires learning unified contact representations from diverse data, yet a fundamental barrier remains: diverse real-world tactile data are prohibitive at scale, while direct zero-shot sim-to-real transfer is challenging due to the complex dynamics of nonlinear deformation of soft sensors. To address this, we propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates. We enable this via a two-stage Sim-to-Real Contact Learning Pipeline: first, we pre-train on a large simulation data set to learn general contact physics; second, we fine-tune on a small set of real data, pseudo-labeled via geometric heuristics and force optimization, to align sensor characteristics. This allows physical generalization to unseen tools. We leverage SCFields as the dense observation input for a diffusion policy to enable robust execution of contact-rich tool manipulation tasks. Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization, significantly outperforming vision-only and raw-tactile baselines.

Related papers

• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning [6.509572742369941]
  We propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand.<n>We introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals.<n>Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining.<n>Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-10T07:46:38Z)
• TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation [50.608989079323784]
  We propose TouchThinker, a tactile-language framework that scales tactile commonsense reasoning to the open world from both data and representation perspectives.<n>First, we construct TouchThinker-1M, a million-scale, multi-source tactile reasoning dataset covering textbf415 objects, textbf8 scenarios, and textbf7 sensor types.<n>Then, we propose action-aware modeling mechanism to improve tactile representation efficiency and enable efficient reasoning.
  arXiv  Detail & Related papers  (2026-06-10T03:58:32Z)
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation [51.463992799813816]
  A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data.<n>We introduce Center-of-Pressure, an effective tactile representation grounded in physical principles.<n>We evaluate CoP on two blind, challenging contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [60.609604885961716]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map [4.188018133636475]
  Vision-Based Tactile Sensors are essential for achieving dexterous robotic manipulation.<n>Current tactile simulations suffer from a persistent dilemma: simplified geometric projections lack physical authenticity.<n>We present Tacmap, a high-fidelity, computationally efficient tactile simulation framework anchored in penetration depth.
  arXiv  Detail & Related papers  (2026-02-25T06:40:59Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation [74.41728218960465]
  We propose a real-to-real 3D data generation framework (R2RGen) that directly augments the pointcloud observation-action pairs to generate real-world data.<n>R2RGen substantially enhances data efficiency on extensive experiments and demonstrates strong potential for scaling and application on mobile manipulation.
  arXiv  Detail & Related papers  (2025-10-09T17:55:44Z)
• Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting [25.355424080824996]
  Tactile sensing is critical for human perception and underpins applications in computer vision, robotics, and multimodal learning.<n>Because tactile data is often scarce and costly to acquire, generating synthetic tactile images provides a scalable solution to augment real-world measurements.<n>We demonstrate that overlapping training and test samples in commonly used datasets inflate performance metrics, obscuring the true generalizability of tactile models.
  arXiv  Detail & Related papers  (2025-03-10T02:37:22Z)
• CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects [1.3299507495084417]
  Nonprehensile manipulation is essential for manipulating objects that are too thin, large, or otherwise ungraspable in the wild. We propose a novel contact-based object representation and pretraining pipeline to tackle this.
  arXiv  Detail & Related papers  (2024-03-16T01:47:53Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Optical Tactile Sim-to-Real Policy Transfer via Real-to-Sim Tactile Image Translation [21.82940445333913]
  We present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images. This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer.
  arXiv  Detail & Related papers  (2021-06-16T13:58:35Z)
• Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing [1.9981375888949469]
  This paper focuses on a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. A strategy is proposed to train a tailored deep neural network entirely from the simulation data. The resulting learning architecture is directly transferable across multiple tactile sensors without further training and yields accurate predictions on real data.
  arXiv  Detail & Related papers  (2020-03-05T14:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.