ReTac-ACT: A State-Gated Vision-Tactile Fusion Transformer for Precision Assembly

• URL: http://arxiv.org/abs/2603.09565v2
• Date: Wed, 18 Mar 2026 12:10:16 GMT
• Title: ReTac-ACT: A State-Gated Vision-Tactile Fusion Transformer for Precision Assembly
• Abstract summary: Precision assembly requires sub-millimeter corrections in contact-rich "last-millimeter" regions.<n>ReTac-ACT addresses this challenge through three mechanisms.<n>It achieves 90%-in-hole success, substantially outperforming vision-only and generic methods, and maintains success at 0.1mm peg.
• Score: 10.687495099840659
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Precision assembly requires sub-millimeter corrections in contact-rich "last-millimeter" regions where visual feedback fails due to occlusion from the end-effector and workpiece. We present ReTac-ACT (Reconstruction-enhanced Tactile ACT), a vision-tactile imitation learning policy that addresses this challenge through three synergistic mechanisms: (i) bidirectional cross-attention enabling reciprocal visuo-tactile feature enhancement before fusion, (ii) a proprioception-conditioned gating network that dynamically elevates tactile reliance when visual occlusion occurs, and (iii) a tactile reconstruction objective enforcing learning of manipulation-relevant contact information rather than generic visual textures. Evaluated on the standardized NIST Assembly Task Board M1 benchmark, ReTac-ACT achieves 90% peg-in-hole success, substantially outperforming vision-only and generalist baseline methods, and maintains 80% success at industrial-grade 0.1mm clearance. Ablation studies validate that each architectural component is indispensable. The ReTac-ACT codebase and a vision-tactile demonstration dataset covering various clearance levels with both visual and tactile features will be released to support reproducible research.

Related papers

• Restore What Matters: Lessons from Joint Restoration and Recognition [72.91692240810536]
  Recognition pipelines typically adopt a restore-then-recognize workflow, yet decades of experience show that generating visually pleasing images seldom to improved recognition.<n>We propose a Joint Restoration-for-Recognition paradigm: restore only what downstream tasks truly require, with task signals dictating where, how much, and whether restoration is necessary.<n>We conclude that better-looking images are neither necessary nor sufficient; restoration modules must be task-driven, selective, and physically aware.
  arXiv  Detail & Related papers  (2026-09-12T08:08:08Z)
• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• ViTaR: Visuo-Tactile Residual Adaptation for Foundation VLA Manipulation [4.374415463111511]
  We introduce ViTaR, which reframes tactile feedback from an action-generating perceptual input to an execution modulator.<n>On the UniVTAC benchmark spanning seven contact-rich tasks, ViTaR achieves 61.3% average success, a 30.6 percentage-point improvement over its frozen VLA base.
  arXiv  Detail & Related papers  (2026-08-16T15:46:11Z)
• AdaDexGrasp: Adaptive Dexterous Grasping via 3D Visuo-Tactile Representation Fusion [60.79507611737292]
  Humans achieve stable and adaptive grasps by seamlessly integrating visual perception and tactile feedback.<n>Existing robotic grasping approaches rely on visual inputs and lack mechanisms for tactile-guided adaptation after contact.<n>We propose a unified visuo-tactile-fusion grasping framework that integrates grasp generation, feasibility prediction, and adaptive refinement.
  arXiv  Detail & Related papers  (2026-08-06T16:29:11Z)
• Tac-DINO: Learning Vision-Tactile Features with Patch Alignment [47.4807111947232]
  We build a data collection system to acquire a large-scale tactile dataset, with over 20 K tactile contacts from 505 real-world objects.<n>We then design a Vis-Tac Holographic Matching Benchmark to evaluate vision-tactile local-to-global alignment ability.<n>Then we propose Vision-Tactile Patch Alignment (VTPA) methods for vision-tactile representation learning.
  arXiv  Detail & Related papers  (2026-06-10T13:33:42Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [60.609604885961716]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• Touch2Insert: Zero-Shot Peg Insertion by Touching Intersections of Peg and Hole [15.791523980428538]
  We present emphTouch2Insert, a tactile-based framework for arbitrary peg insertion.<n>Our method reconstructs cross-sectional geometry from high-resolution tactile images and estimates the relative pose of the hole with respect to the peg in a zero-shot manner.<n>Results indicate that Touch2Insert achieved sub-millimeter pose estimation accuracy for all connectors in simulation, and attained an average success rate of 86.7% on the real robot.
  arXiv  Detail & Related papers  (2026-03-04T01:35:33Z)
• Symmetry-Aware Fusion of Vision and Tactile Sensing via Bilateral Force Priors for Robotic Manipulation [7.104060092661104]
  We propose a Cross-Modal Transformer (CMT) for visuo-tactile fusion.<n>CMT integrates wrist-camera observations with tactile signals through structured self- and cross-attention.<n>Experiments on the TacSL benchmark show that CMT with symmetry regularization achieves a 96.59% insertion success rate.
  arXiv  Detail & Related papers  (2026-02-14T09:19:48Z)
• UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking [47.900334665380115]
  We propose UniVTAC, a simulation-based visuo-tactile data platform.<n>We introduce the UniVTAC encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data.<n>We present the UniVTAC Benchmark, which consists of eight representative visuo-tactile manipulation tasks.
  arXiv  Detail & Related papers  (2026-02-10T18:57:00Z)
• Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities [19.45726946555448]
  Tactile sensing offers rich and complementary information to vision and language, enabling robots to perceive fine-grained object properties.<n>Existing methods fail to fully integrate the intermediate communication among tactile, language, and vision modalities.<n>We propose TLV-CoRe, a CLIP-based Tactile-Language-Vision Collaborative Representation learning method.
  arXiv  Detail & Related papers  (2025-11-14T17:34:20Z)
• ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations [7.870120920732663]
  We propose ConViTac, a visual-tactile representation learning network designed to enhance the alignment of features during fusion.<n>Our key contribution is a Contrastive Embedding Conditioning mechanism that leverages a contrastive encoder pretrained to project visual and tactile inputs into unified latent embeddings.<n>We conduct extensive experiments to demonstrate the superiority of ConViTac in real world over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-25T18:43:35Z)
• Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting [25.355424080824996]
  Tactile sensing is critical for human perception and underpins applications in computer vision, robotics, and multimodal learning.<n>Because tactile data is often scarce and costly to acquire, generating synthetic tactile images provides a scalable solution to augment real-world measurements.<n>We demonstrate that overlapping training and test samples in commonly used datasets inflate performance metrics, obscuring the true generalizability of tactile models.
  arXiv  Detail & Related papers  (2025-03-10T02:37:22Z)
• Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor [14.492202828369127]
  We leverage a multimodal visuotactile sensor within the framework of imitation learning (IL) to perform contact-rich tasks.<n>We introduce two algorithmic contributions, tactile force matching and learned mode switching, as complimentary methods for improving IL.<n>Our results show that the inclusion of force matching raises average policy success rates by 62.5%, visuotactile mode switching by 30.3%, and visuotactile data as a policy input by 42.5%.
  arXiv  Detail & Related papers  (2023-11-02T14:02:42Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Active 3D Shape Reconstruction from Vision and Touch [66.08432412497443]
  Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. In 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings. We introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile priors to guide the shape exploration; and 3) a set of data-driven solutions with either tactile or visuo
  arXiv  Detail & Related papers  (2021-07-20T15:56:52Z)
• Generative Partial Visual-Tactile Fused Object Clustering [81.17645983141773]
  We propose a Generative Partial Visual-Tactile Fused (i.e., GPVTF) framework for object clustering. A conditional cross-modal clustering generative adversarial network is then developed to synthesize one modality conditioning on the other modality. To the end, two pseudo-label based KL-divergence losses are employed to update the corresponding modality-specific encoders.
  arXiv  Detail & Related papers  (2020-12-28T02:37:03Z)
• 3D Shape Reconstruction from Vision and Touch [62.59044232597045]
  In 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. We introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects.
  arXiv  Detail & Related papers  (2020-07-07T20:20:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.