PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation

• URL: http://arxiv.org/abs/2603.04531v1
• Date: Wed, 04 Mar 2026 19:17:42 GMT
• Title: PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation
• Abstract summary: We introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation.<n>We demonstrate that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing.
• Score: 43.81692375130905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imitation learning, obtaining high quality demonstrations for multi-fingered hands via robot teleoperation or kinesthetic teaching is prohibitive. Alternatively, with reinforcement we can learn skills in simulation, but fast and realistic simulation of tactile observations is challenging. To bridge this gap, we introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation. Instead of simulating tactile sensors or relying purely on proprioceptive policies to transfer zero-shot sim-to-real, our key idea is to leverage privileged sensors in the real world to collect real-world tactile policy data. This data is then used to distill a robust state estimator that operates on tactile input. We demonstrate from our experiments that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing. On the benchmark in-hand rotation task, PTLD achieves a 182% improvement over a proprioception only policy. We also show that PTLD enables learning the challenging task of tactile in-hand reorientation where we see a 57% improvement in the number of goals reached over using proprioception alone. Website: https://akashsharma02.github.io/ptld-website/.

Related papers

• Imagining the Sense of Touch: Touch-Informed Manipulation via Imagined Tactile Representations [4.954401349140998]
  We present TacImag, a tactile imagination framework that predicts tactile observations from vision and proprioception and uses the generated signals to guide manipulation policies.<n>We evaluate TacImag on six simulated and four real-world manipulation tasks.<n>In real-world experiments, imagined force fields improve contact-sensitive tasks by 44.4% on average, whereas imagined tactile images improve texture-sensitive tasks by 23.3%.
  arXiv  Detail & Related papers  (2026-07-02T04:27:11Z)
• T-Rex: Tactile-Reactive Dexterous Manipulation [139.71263755530654]
  We introduce a variable-rate Mixture-of-Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder.<n>We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation.
  arXiv  Detail & Related papers  (2026-06-15T17:59:55Z)
• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning [6.509572742369941]
  We propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand.<n>We introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals.<n>Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining.<n>Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-10T07:46:38Z)
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation [51.463992799813816]
  A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data.<n>We introduce Center-of-Pressure, an effective tactile representation grounded in physical principles.<n>We evaluate CoP on two blind, challenging contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos [56.510263910611684]
  We tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions.<n>Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors.<n>We present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data.
  arXiv  Detail & Related papers  (2026-02-13T18:59:10Z)
• Enhancing Tactile-based Reinforcement Learning for Robotic Control [32.565866574593635]
  We develop self-supervised learning (SSL) methodologies to more effectively harness tactile observations.<n>We empirically demonstrate that sparse binary tactile signals are critical for dexterity.<n>We release the Robot Tactile Olympiad (RoTO) benchmark to standardise and promote future research in tactile-based manipulation.
  arXiv  Detail & Related papers  (2025-10-24T16:15:05Z)
• VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning [39.49846628626501]
  Humans excel at bimanual assembly tasks by adapting to rich tactile feedback.<n>We present VT-Refine, a visuo-tactile policy learning framework that combines real-world demonstrations, high-fidelity tactile simulation, and reinforcement learning.
  arXiv  Detail & Related papers  (2025-10-16T17:41:36Z)
• Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids [56.892520712892804]
  We introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three dexterous manipulation tasks.<n>We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors.
  arXiv  Detail & Related papers  (2025-02-27T18:59:52Z)
• Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensing [43.269672740168396]
  We introduce a sensor model for tactile skin that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces.<n>We conduct extensive real-world experiments to assess how tactile sensing facilitates policy adaptation to various unseen object properties.
  arXiv  Detail & Related papers  (2024-07-10T17:52:30Z)
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality [64.51295032956118]
  We train a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups.
  arXiv  Detail & Related papers  (2022-10-25T01:51:36Z)
• Sim2Real for Peg-Hole Insertion with Eye-in-Hand Camera [58.720142291102135]
  We use a simulator to learn the peg-hole insertion problem and then transfer the learned model to the real robot. We show that the transferred policy, which only takes RGB-D and joint information (proprioception) can perform well on the real robot.
  arXiv  Detail & Related papers  (2020-05-29T05:58:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.