Tactile Sim2Real without Tactile Simulation via Bottlenecked Latent Reconstruction

• URL: http://arxiv.org/abs/2608.15897v2
• Date: Wed, 19 Aug 2026 00:46:03 GMT
• Title: Tactile Sim2Real without Tactile Simulation via Bottlenecked Latent Reconstruction
• Abstract summary: Sim2Real avoids sensor-specific simulation entirely by training policies on a simulator-native oracle sensor.<n>We show that SBLR matches or approaches the performance of an oracle with direct access to tactile simulation.
• Score: 11.374127908457446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robot sensor designs, particularly tactile sensors, are highly diverse and evolve rapidly. Modeling each sensor in simulation demands substantial domain expertise and computational approximations can degrade the fidelity of the simulated signals. We propose Sim2Real via Bottlenecked Latent Reconstruction (SBLR), a framework that avoids sensor-specific simulation entirely by (1) training policies on a simulator-native oracle sensor that is easy to construct without modeling any particular sensor (e.g. we use a point-cloud and finger-tip forces as a tactile oracle), and (2) aligning real sensor latent embeddings to those of the oracle sensor at inference time. Policy training proceeds in two-stage: the policy first learns from the oracle sensor latents, then a bottlenecked latent reconstruction adapts it to the information loss expected when using the real sensor instead of the oracle. The alignment between oracle and real sensor is learned from unpaired random-play data collected in both simulation and the real world, using rectified-flow-based transformation networks trained on nearest-neighbor pseudo-pairs. Simulation experiments on three contact-rich tasks show that SBLR matches or approaches the performance of an oracle with direct access to tactile simulation. Hardware experiments on Peg Insertion and Gear Meshing with GelSight Mini and DIGIT sensors demonstrate 85-97.5% zero-shot success without requiring any sensor-specific modeling or calibration, outperforming a physics-based tactile simulation baseline by 7.5-15%.

Related papers

• Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration [50.18488308004802]
  Sen-Cap is a 3D motion capture framework that integrates multi-modal data from LiDAR and camera.<n>Sen-Cap achieves state-of-the-art performance on major metrics on Human-M3 and FreeMotion.<n>This combination of flexibility and robustness opens new opportunities for motion capture in real-world scenarios.
  arXiv  Detail & Related papers  (2026-08-03T14:16:18Z)
• TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer [8.357674069719524]
  We propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space.<n>Our approach employs modality-specific encoders to project diverse tactile observations into a common embedding space.<n>Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations.
  arXiv  Detail & Related papers  (2026-06-17T11:41:27Z)
• DOT-Sim: Differentiable Optical Tactile Simulation with Precise Real-to-Sim Physical Calibration [12.64545897591664]
  We propose DOT-Sim: Differentiable Optical Tactile Simulation.<n> DOT-Sim accurately captures the physical behavior of soft sensors by modeling them as elastic materials.<n>To handle the optical aspect, we propose a novel approach to simulating optical responses by learning a residual image.
  arXiv  Detail & Related papers  (2026-04-30T03:27:09Z)
• SPLIT: Separating Physical-Contact via Latent Arithmetic in Image-Based Tactile Sensors [0.5500249707065662]
  This paper presents SPLIT, a novel method for simulating image-based tactile sensors.<n>Central to our approach is a latent space arithmetic strategy that explicitly disentangles contact geometry from sensor-specific optical properties.<n>We provide a calibrated finite element method (FEM) soft-body mesh simulation with variable resolution, offering a tunable trade-off between speed and fidelity.
  arXiv  Detail & Related papers  (2026-04-27T13:13:58Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• A Foundation Model for Virtual Sensors [41.4965371110973]
  Existing virtual sensor approaches require application-specific models with hand-selected inputs for each sensor.<n>We introduce the first foundation model for virtual sensors addressing both limitations.<n>Our architecture achieves 415x reduction in time and 951x reduction in memory requirements.
  arXiv  Detail & Related papers  (2026-01-28T14:17:46Z)
• TwinTac: A Wide-Range, Highly Sensitive Tactile Sensor with Real-to-Sim Digital Twin Sensor Model [4.668679765548824]
  We present TwinTac, a system that combines the design of a physical tactile sensor with its digital twin model.<n>Our hardware sensor is designed for high sensitivity and a wide measurement range.<n>We show that simulation data generated by our digital twin sensor can effectively augment real-world data, leading to improved accuracy.
  arXiv  Detail & Related papers  (2025-09-12T08:51:28Z)
• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation [34.47272224723296]
  We present Taccel, a high-performance simulation platform that integrates IPC and ABD to model robots, tactile sensors, and objects with both accuracy and unprecedented speed.<n>Unlike previous simulators that operate at sub-real-time speeds with limited parallelization, Taccel provides precise physics simulation and realistic tactile signals.<n>These capabilities position Taccel as a powerful tool for scaling up tactile robotics research and development, potentially transforming how robots interact with and understand their physical environment.
  arXiv  Detail & Related papers  (2025-04-17T12:57:11Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• Bayesian Imitation Learning for End-to-End Mobile Manipulation [80.47771322489422]
  Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities. We show that using the Variational Information Bottleneck to regularize convolutional neural networks improves generalization to held-out domains. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities.
  arXiv  Detail & Related papers  (2022-02-15T17:38:30Z)
• Real-time detection of uncalibrated sensors using Neural Networks [62.997667081978825]
  An online machine-learning based uncalibration detector for temperature, humidity and pressure sensors was developed. The solution integrates an Artificial Neural Network as main component which learns from the behavior of the sensors under calibrated conditions. The obtained results show that the proposed solution is able to detect uncalibrations for deviation values of 0.25 degrees, 1% RH and 1.5 Pa, respectively.
  arXiv  Detail & Related papers  (2021-02-02T15:44:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.