TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer

• URL: http://arxiv.org/abs/2606.18959v1
• Date: Wed, 17 Jun 2026 11:41:27 GMT
• Title: TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer
• Abstract summary: We propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space.<n>Our approach employs modality-specific encoders to project diverse tactile observations into a common embedding space.<n>Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations.
• Score: 8.357674069719524
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Tactile sensing provides direct measurements of contact interactions that are essential for robotic manipulation. However, current simulators lack the fidelity to faithfully model the complex deformation and transduction mechanics of tactile sensors, severely hindering sim-to-real transfer in robot learning pipelines. To address this challenge, we propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space, eliminating the need for accurate raw-signal simulation while preserving relevant contact information. Our approach employs modality-specific encoders to project diverse tactile observations, such as simulated penetration depth and real-world capacitance, into a common embedding space. The model is trained using self- and cross-reconstruction objectives alongside contrastive alignment, encouraging modality-invariant yet information-rich representations. We evaluate the learned embeddings on indenter shape identification, force prediction, and geometric reconstruction tasks, training exclusively in simulation and testing directly on real sensor measurements. Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations. Furthermore, incorporating multi-physics simulation modalities yields more informative embeddings that transfer across diverse downstream tasks, demonstrating a 16.7% reduction in force prediction error and a 45.8% reduction in shape reconstruction error. Finally, we release an efficient Warp-based implementation of a penalty-based tactile simulation model for Isaac Lab, enabling scalable tactile data generation.

Related papers

• Tactile Sim2Real without Tactile Simulation via Bottlenecked Latent Reconstruction [11.374127908457446]
  Sim2Real avoids sensor-specific simulation entirely by training policies on a simulator-native oracle sensor.<n>We show that SBLR matches or approaches the performance of an oracle with direct access to tactile simulation.
  arXiv  Detail & Related papers  (2026-08-16T19:09:55Z)
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation [51.463992799813816]
  A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data.<n>We introduce Center-of-Pressure, an effective tactile representation grounded in physical principles.<n>We evaluate CoP on two blind, challenging contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds [36.87909899472009]
  We introduce SIM1, a physics-aligned real-to-sim-to-real data engine that grounds simulation in the physical world.<n>The system digitizes scenes into metric-consistent twins, calibrates deformable dynamics through elastic modeling, and expands behaviors via diffusion-based trajectory generation.<n>Experiments show that policies trained on purely synthetic data achieve parity with real-data baselines at a 1:15 equivalence ratio, while delivering 90% zero-shot success and 50% generalization gains in real-world deployment.
  arXiv  Detail & Related papers  (2026-04-09T17:59:52Z)
• Few-Shot Neural Differentiable Simulator: Real-to-Sim Rigid-Contact Modeling [9.084754731100311]
  We propose a few-shot real-to-sim approach that combines analytical formulations with the representational capacity of graph neural network (GNN)-based models.<n>Using only a small amount of real-world data, our method calibrates analytical simulators to generate large-scale synthetic datasets that capture diverse contact interactions.<n> Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories.
  arXiv  Detail & Related papers  (2026-03-06T12:32:56Z)
• D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping [66.22412592525369]
  We introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine.<n>We show that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values.<n>Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping.
  arXiv  Detail & Related papers  (2026-03-01T15:32:04Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• Data assimilation and discrepancy modeling with shallow recurrent decoders [2.47593085771929]
  We propose a machine learning framework for Data Assimilation with a SHallow REcurrent Decoder (DA-SHRED)<n>Our algorithm incorporates a sparse identification of nonlinear dynamics based regression model in the latent space to identify functionals corresponding to missing dynamics in the simulation model.<n>We demonstrate that DA-SHRED successfully closes the SIM2REAL gap and additionally recovers missing dynamics in highly complex systems.
  arXiv  Detail & Related papers  (2025-12-01T01:01:48Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)
• Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing [1.9981375888949469]
  This paper focuses on a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. A strategy is proposed to train a tailored deep neural network entirely from the simulation data. The resulting learning architecture is directly transferable across multiple tactile sensors without further training and yields accurate predictions on real data.
  arXiv  Detail & Related papers  (2020-03-05T14:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.