Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation

• URL: http://arxiv.org/abs/2605.28812v1
• Date: Wed, 27 May 2026 17:59:02 GMT
• Title: Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation
• Abstract summary: A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data.<n>We introduce Center-of-Pressure, an effective tactile representation grounded in physical principles.<n>We evaluate CoP on two blind, challenging contact-rich manipulation tasks.
• Score: 51.463992799813816
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prevents information-dense modalities like touch from being effectively used. Existing sim-to-real methods often mitigate this gap by simplifying tactile data into coarse low-dimensional features -- sacrificing the richness required for complex manipulation. In this work, we introduce Center-of-Pressure (CoP), an effective tactile representation grounded in physical principles that preserves dense contact information while maintaining robustness for sim-to-real transfer. To support this representation, we propose a sensor calibration scheme based on differentiable dynamics, enabling the estimation of taxel orientations without requiring ground-truth force measurements. We evaluate CoP on two blind, challenging contact-rich manipulation tasks: peg-in-hole insertion and ball balancing. Across both tasks, policies conditioned on CoP achieve zero-shot sim-to-real transfer on a multi-fingered hand, and outperform both coarse binary-contact and raw-taxel baselines. Analysis of learned policy states further suggests that CoP-conditioned policies encode task-relevant physical properties, such as object mass, as an emergent byproduct of control.

Related papers

• ConCent: Contact-Centric Real-to-Sim-to-Real Learning from One Demonstration [17.067367964368486]
  Sim-to-real policy transfer is a promising paradigm for scaling robot manipulation without large-scale real-world data.<n>We propose a contact-centric real-to-sim-to-real RL framework that uses task-relevant contact event sequences extracted from real demonstrations.<n>Experiments on contact-rich manipulation tasks demonstrate more stable and robust sim-to-real policy transfer compared to unconstrained RL baselines.
  arXiv  Detail & Related papers  (2026-06-29T13:16:33Z)
• Direction Matters: Learning Force Direction Enables Sim-to-Real Contact-Rich Manipulation [28.707088673008172]
  We propose a framework that leverages expert-designed controller logic for transfer.<n>Inspired by the success of privileged supervision in tasks, we employ a human-designed finite state machine based position/force controller in simulation.<n>Unlike force magnitudes, which are highly sensitive to simulation inaccuracies, force directions encode high-level task geometry and remain robust across the sim-to-real gap.
  arXiv  Detail & Related papers  (2026-02-15T14:57:13Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting [60.513168855890974]
  Cross-domain transfer in robotic manipulation is a longstanding challenge due to the significant domain gap between simulated and real-world environments.<n>We propose Semantic 2D Gaussian Splatting (S2GS), a novel representation method that extracts object-centric, domain-invariant spatial features.<n>S2GS significantly improves sim-to-real transferability, maintaining high and stable task performance in real-world scenarios.
  arXiv  Detail & Related papers  (2025-12-04T12:16:15Z)
• Flow with the Force Field: Learning 3D Compliant Flow Matching Policies from Force and Demonstration-Guided Simulation Data [15.490144903271338]
  We introduce a framework for generating force-informed data in simulation, instantiated by a single human demonstration.<n>We show how coupling with a compliant policy improves the performance of a visuomotor policy learned from synthetic data.
  arXiv  Detail & Related papers  (2025-10-03T05:40:58Z)
• Multi-Modal Manipulation via Multi-Modal Policy Consensus [62.49978559936122]
  We propose a new approach to integrate diverse sensory modalities for robotic manipulation.<n>Our method factorizes the policy into a set of diffusion models, each specialized for a single representation.<n>We evaluate our approach on simulated manipulation tasks in RLBench, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion.
  arXiv  Detail & Related papers  (2025-09-27T19:43:04Z)
• Dynamic Manipulation of Deformable Objects in 3D: Simulation, Benchmark and Learning Strategy [88.8665000676562]
  Prior methods often simplify the problem to low-speed or 2D settings, limiting their applicability to real-world 3D tasks.<n>To mitigate data scarcity, we introduce a novel simulation framework and benchmark grounded in reduced-order dynamics.<n>We propose Dynamics Informed Diffusion Policy (DIDP), a framework that integrates imitation pretraining with physics-informed test-time adaptation.
  arXiv  Detail & Related papers  (2025-05-23T03:28:25Z)
• Optical Tactile Sim-to-Real Policy Transfer via Real-to-Sim Tactile Image Translation [21.82940445333913]
  We present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images. This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer.
  arXiv  Detail & Related papers  (2021-06-16T13:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.