Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assembly

• URL: http://arxiv.org/abs/2609.07534v2
• Date: Mon, 14 Sep 2026 04:13:47 GMT
• Title: Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assembly
• Abstract summary: We present PACE (Proprioception-Anchored Cross-Modalception), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions.<n>Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3% and only a 2.7-percentage-point sim-to-real drop.
• Score: 9.415285027294221
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contact-rich assembly remains challenging because it requires submillimeter spatial accuracy and reliable interpretation of forces during sustained contact. Although simulation-based reinforcement learning offers a scalable training paradigm, discrepancies in visual observations, contact dynamics, and force/torque (F/T) measurements often limit policy transfer. We observe that proprioception is comparatively consistent across domains because joint positions are expressed in a shared calibrated coordinate system and joint velocities are computed consistently in simulation and on hardware. Based on this observation, we present PACE (Proprioception-Anchored Cross-Modal Encoder), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions. Static domain-specific factors, including lighting, texture, and sensor bias, contain little information about joint motion; the proposed objective therefore encourages the encoder to suppress these factors while retaining task-relevant motion cues. Policies trained on frozen PACE features are deployed on hardware without real-world fine-tuning or object-pose tracking. Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3\% and only a 2.7-percentage-point sim-to-real drop, while remaining robust to perturbations that substantially degrade pose-based and learned-fusion baselines.

Related papers

• CARO: Contact-Agnostic Residual Observation for Zero-Shot Robust Quadruped Locomotion [6.612698611142421]
  CARO is a contact-agnostic residual observation framework for policy adaptation.<n>It embeds a fixed-base Euler--Lagrange model into a reinforcement learning control loop and constructs a torque-level residual observation.<n>A disturbance observer extracts a structured signal representing dynamics mismatch, while the policy learns to exploit this feedback for online adaptation.
  arXiv  Detail & Related papers  (2026-08-25T08:23:24Z)
• TRACE: Learned Proprioceptive Odometry for Legged Robots under Unreliable Contact Conditions [4.317571381146645]
  We present TRACE (Tokenized Robust Attention for Contact-Aware Estimation), an end-to-end learned proprioceptive odometry estimator for legged robots.<n>The proposed estimator directly predicts relative displacement, relative rotation, and body-frame velocity from a recent history of onboard inertial and joint measurements.<n>To improve robustness under unreliable contact conditions, we introduce a foot-aware cross-attention module that adaptively weights IMU and leg-wise kinematic tokens.
  arXiv  Detail & Related papers  (2026-08-06T12:53:33Z)
• A Task-Space Receding Horizon Controller for Fast Collision Avoidance [0.08594140167290099]
  Real-time collision avoidance for robotic manipulators requires fast reactions to unexpected obstacle motion.<n>Full model predictive control can provide this foresight, but its online cost may grow quickly with horizon length, model fidelity, and the number of active geometric constraints.<n>We present a task-space receding-horizon controller that uses a short contact-consistent rollout to generate a terminal kinematic reference.
  arXiv  Detail & Related papers  (2026-07-17T08:13:12Z)
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation [51.463992799813816]
  A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data.<n>We introduce Center-of-Pressure, an effective tactile representation grounded in physical principles.<n>We evaluate CoP on two blind, challenging contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulation [8.276456155150138]
  We present Spacetime Optimal-Transport Attention (SO-TA), a tri-modal fusion backbone that replaces softmax-normalized patch attention.<n>Explicit marginal constraints act as a structured inductive bias for contact-rich tasks, encouraging conditioning-aware spatial selection.<n>We evaluate SO-TA on three real-robot tasks: tight peg-in-hole assembly, BCM wiring-connector insertion, and curved-surface mark erasing.
  arXiv  Detail & Related papers  (2026-05-19T19:29:21Z)
• SECOND-Grasp: Semantic Contact-guided Dexterous Grasping [60.1519218638742]
  Second-Grasp (SEmantic CONtact-guided Dexterous Grasping) is a unified framework that enables robotic hands to adjust grasping strategies based on semantic reasoning.<n>We show that our approach consistently outperforms baselines in lifting success rate on both seen and unseen categories.
  arXiv  Detail & Related papers  (2026-05-13T07:37:00Z)
• Event-based SLAM Benchmark for High-Speed Maneuvers [68.148886127117]
  Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution.<n>Existing event-based approaches, although successful in mitigating motion blur caused by high-speed maneuvers, suffer from many limitations.<n>We introduce a benchmarking framework for event-based state estimation, called EvSLAM, characterized by sufficient variation in data collection platforms.
  arXiv  Detail & Related papers  (2026-04-27T04:36:41Z)
• Hypergraph-State Collaborative Reasoning for Multi-Object Tracking [63.32950991964095]
  We propose a collaborative reasoning framework that enhances motion estimation through joint inference among multiple correlated objects.<n>By allowing objects with similar motion states to mutually constrain and refine each other, our framework stabilizes noisy trajectories and infers plausible motion continuity even when target is occluded.
  arXiv  Detail & Related papers  (2026-04-14T12:37:39Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• ImplicitRDP: An End-to-End Visual-Force Diffusion Policy with Structural Slow-Fast Learning [52.86018040861575]
  We propose a unified end-to-end visual-force diffusion policy that integrates visual planning and reactive force control within a single network.<n>We introduce Structural Slow-Fast Learning, a mechanism utilizing causal attention to simultaneously process asynchronous visual and force tokens.<n>Experiments on contact-rich tasks demonstrate that ImplicitRDP significantly outperforms both vision-only and hierarchical baselines.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• MATE: Motion-Augmented Temporal Consistency for Event-based Point Tracking [58.719310295870024]
  This paper presents an event-based framework for tracking any point.<n>To resolve ambiguities caused by event sparsity, a motion-guidance module incorporates kinematic vectors into the local matching process.<n>The method improves the $Survival_50$ metric by 17.9% over event-only tracking of any point baseline.
  arXiv  Detail & Related papers  (2024-12-02T09:13:29Z)
• Robust Collaborative Perception without External Localization and Clock Devices [52.32342059286222]
  A consistent spatial-temporal coordination across multiple agents is fundamental for collaborative perception. Traditional methods depend on external devices to provide localization and clock signals. We propose a novel approach: aligning by recognizing the inherent geometric patterns within the perceptual data of various agents.
  arXiv  Detail & Related papers  (2024-05-05T15:20:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.