Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulation

• URL: http://arxiv.org/abs/2605.20433v1
• Date: Tue, 19 May 2026 19:29:21 GMT
• Title: Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulation
• Abstract summary: We present Spacetime Optimal-Transport Attention (SO-TA), a tri-modal fusion backbone that replaces softmax-normalized patch attention.<n>Explicit marginal constraints act as a structured inductive bias for contact-rich tasks, encouraging conditioning-aware spatial selection.<n>We evaluate SO-TA on three real-robot tasks: tight peg-in-hole assembly, BCM wiring-connector insertion, and curved-surface mark erasing.
• Score: 8.276456155150138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contact-rich manipulation tasks such as tight-clearance insertion, connector mating, polishing, and surface-conforming wiping remain difficult for data-driven controllers because they couple discontinuous contact dynamics, partial observability, and strict safety constraints. No single sensing modality suffices: vision supplies global context before contact, force/torque (F/T) feedback governs interaction after contact, and proprioceptive pose provides a consistent kinematic backbone. Most prior imitation-learning policies for contact-rich tasks operate on uni- or bi-modal signals, and the few that fuse three modalities typically adopt off-the-shelf attention modules with no explicit prior on how attention mass should be distributed across task-relevant regions. We present Spacetime Optimal-Transport Attention (SO-TA), a tri-modal fusion backbone that replaces softmax-normalized patch attention by an entropy-regularized Optimal Transport (OT) alignment between force-pose-derived sub-queries and visual patches. Explicit marginal constraints act as a structured inductive bias for contact-rich tasks, encouraging conditioning-aware spatial selection that is stable across illumination, distractors, and partial occlusion. SO-TA is paired with a diffusion-based sequence policy mapping observation windows to pose-action chunks. We evaluate SO-TA on three real-robot tasks: tight peg-in-hole assembly, BCM wiring-connector insertion, and curved-surface mark erasing. With ~200 rollouts per condition, SO-TA reaches 100% success on tight peg-in-hole versus 93% for cross-attention at matched capacity, and retains 82.5% success under illumination, distractor, and partial-occlusion perturbations where a concatenation baseline drops to 43.5%. OT-derived patch heatmaps and leave-one-out modality-influence ratios provide interpretable, phase-dependent diagnostics.

Related papers

• Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Contact-Rich Assembly [9.415285027294221]
  We present PACE (Proprioception-Anchored Cross-Modalception), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions.<n>Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3% and only a 2.7-percentage-point sim-to-real drop.
  arXiv  Detail & Related papers  (2026-09-07T14:20:22Z)
• Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation [74.87716678195099]
  Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation.<n>Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge.<n>We propose a motion-aware tactile representation to facilitate contact-rich manipulation.
  arXiv  Detail & Related papers  (2026-06-29T08:20:18Z)
• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation [58.33732779555114]
  We argue that the key bottleneck is a representation-level failure caused by two coupled heterogeneities.<n>We propose Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation (SpectraMB)<n>SpectraMB achieves the best results in most evaluation settings and exhibits improved robustness under noisy interactions.
  arXiv  Detail & Related papers  (2026-06-01T15:58:13Z)
• SECOND-Grasp: Semantic Contact-guided Dexterous Grasping [60.1519218638742]
  Second-Grasp (SEmantic CONtact-guided Dexterous Grasping) is a unified framework that enables robotic hands to adjust grasping strategies based on semantic reasoning.<n>We show that our approach consistently outperforms baselines in lifting success rate on both seen and unseen categories.
  arXiv  Detail & Related papers  (2026-05-13T07:37:00Z)
• ReTac-ACT: A State-Gated Vision-Tactile Fusion Transformer for Precision Assembly [10.687495099840659]
  Precision assembly requires sub-millimeter corrections in contact-rich "last-millimeter" regions.<n>ReTac-ACT addresses this challenge through three mechanisms.<n>It achieves 90%-in-hole success, substantially outperforming vision-only and generic methods, and maintains success at 0.1mm peg.
  arXiv  Detail & Related papers  (2026-03-10T12:09:22Z)
• Symmetry-Aware Fusion of Vision and Tactile Sensing via Bilateral Force Priors for Robotic Manipulation [7.104060092661104]
  We propose a Cross-Modal Transformer (CMT) for visuo-tactile fusion.<n>CMT integrates wrist-camera observations with tactile signals through structured self- and cross-attention.<n>Experiments on the TacSL benchmark show that CMT with symmetry regularization achieves a 96.59% insertion success rate.
  arXiv  Detail & Related papers  (2026-02-14T09:19:48Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• A Unified Complementarity-based Approach for Rigid-Body Manipulation and Motion Prediction [12.241353619398899]
  This paper presents a mathematical model for robotic manipulation that consistently captures both free motion and frictional contact within a single formalism (Unicomp)<n> Experimental results show that the proposed approach enables physically consistent behavior at interactive speeds across tasks, from pushing to contact-rich whole-body maneuvers.
  arXiv  Detail & Related papers  (2026-02-04T13:10:57Z)
• Towards Efficient 3D Object Detection for Vehicle-Infrastructure Collaboration via Risk-Intent Selection [23.186947629238233]
  Risk-intent Selective detection (RiSe) is an interaction-aware framework that shifts the paradigm from identifying visible regions to prioritizing risk-critical ones.<n>RiSe implements a semantic-selective fusion scheme that transmits high-fidelity features only from high-interaction regions.<n>Our method reduces communication volume to 0.71% of full feature sharing while maintaining state-of-the-art detection accuracy.
  arXiv  Detail & Related papers  (2026-01-06T13:25:23Z)
• ImplicitRDP: An End-to-End Visual-Force Diffusion Policy with Structural Slow-Fast Learning [52.86018040861575]
  We propose a unified end-to-end visual-force diffusion policy that integrates visual planning and reactive force control within a single network.<n>We introduce Structural Slow-Fast Learning, a mechanism utilizing causal attention to simultaneously process asynchronous visual and force tokens.<n>Experiments on contact-rich tasks demonstrate that ImplicitRDP significantly outperforms both vision-only and hierarchical baselines.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models [55.30555646945055]
  Text-to-Image (T2I) models are vulnerable to semantic leakage.<n>We introduce DeLeaker, a lightweight approach that mitigates leakage by directly intervening on the model's attention maps.<n>SLIM is the first dataset dedicated to semantic leakage.
  arXiv  Detail & Related papers  (2025-10-16T17:39:21Z)
• Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding [53.377028000325424]
  We propose an Iterative Alignment Network (IA-Net) for temporal sentence grounding task. We pad multi-modal features with learnable parameters to alleviate the nowhere-to-attend problem of non-matched frame-word pairs. We also devise a calibration module following each attention module to refine the alignment knowledge.
  arXiv  Detail & Related papers  (2021-09-14T02:08:23Z)
• More Than Just Attention: Learning Cross-Modal Attentions with Contrastive Constraints [63.08768589044052]
  We propose Contrastive Content Re-sourcing ( CCR) and Contrastive Content Swapping ( CCS) constraints to address such limitation. CCR and CCS constraints supervise the training of attention models in a contrastive learning manner without requiring explicit attention annotations. Experiments on both Flickr30k and MS-COCO datasets demonstrate that integrating these attention constraints into two state-of-the-art attention-based models improves the model performance.
  arXiv  Detail & Related papers  (2021-05-20T08:48:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.