Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly

• URL: http://arxiv.org/abs/2603.20290v1
• Date: Wed, 18 Mar 2026 14:58:17 GMT
• Title: Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly
• Abstract summary: We propose a general transparent fragments contour estimation framework based on visual-tactile fusion.<n>By fusing tactile information with visual cues, a visual-tactile fusion material is proposed.<n> experimental results demonstrate the validity of the proposed framework.
• Score: 69.97596838714595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The contour estimation of transparent fragments is very important for autonomous reassembly, especially in the fields of precision optical instrument repair, cultural relic restoration, and identification of other precious device broken accidents. Different from general intact transparent objects, the contour estimation of transparent fragments face greater challenges due to strict optical properties, irregular shapes and edges. To address this issue, a general transparent fragments contour estimation framework based on visual-tactile fusion is proposed in this paper. First, we construct the transparent fragment dataset named TransFrag27K, which includes a multiscene synthetic data of broken fragments from multiple types of transparent objects, and a scalable synthetic data generation pipeline. Secondly, we propose a visual grasping position detection network named TransFragNet to identify, locate and segment the sampling grasping position. And, we use a two-finger gripper with Gelsight Mini sensors to obtain reconstructed tactile information of the lateral edge of the fragments. By fusing this tactile information with visual cues, a visual-tactile fusion material classifier is proposed. Inspired by the way humans estimate a fragment's contour combining vision and touch, we introduce a general transparent fragment contour estimation framework based on visual-tactile fusion, demonstrates strong performance in real-world validation. Finally, a multi-dimensional similarity metrics based contour matching and reassembly algorithm is proposed, providing a reproducible benchmark for evaluating visual-tactile contour estimation and fragment reassembly. The experimental results demonstrate the validity of the proposed framework. The dataset and codes are available at https://github.com/Keithllin/Transparent-Fragments-Contour-Estimation.

Related papers

• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• DiffTrans: Differentiable Geometry-Materials Decomposition for Reconstructing Transparent Objects [53.83670041249326]
  Reconstructing transparent objects from a set of multi-view images is a challenging task due to the complicated nature and indeterminate behavior of light propagation.<n>We propose a differentiable rendering framework for transparent objects, dubbed DiffTrans, which allows for efficient decomposition and reconstruction of the geometry and materials of transparent objects.
  arXiv  Detail & Related papers  (2026-02-28T02:21:31Z)
• Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects [1.9735804580163874]
  This paper presents a novel cross-modal visuo-tactile perception framework for the 3D shape reconstruction of deformable linear objects (DLOs)<n>Unlike existing methods relying predominantly on vision, the proposed approach integrates foundation-model-based visual perception with adaptive exploration.<n>Results show that the framework accurately reconstructs both simple and highly curved single or multiple cable configurations, even when large portions are occluded.
  arXiv  Detail & Related papers  (2026-01-20T13:57:59Z)
• TransBridge: Boost 3D Object Detection by Scene-Level Completion with Transformer Decoder [66.22997415145467]
  This paper presents a joint completion and detection framework that improves the detection feature in sparse areas.<n> Specifically, we propose TransBridge, a novel transformer-based up-sampling block that fuses the features from the detection and completion networks.<n>The results show that our framework consistently improves end-to-end 3D object detection, with the mean average precision (mAP) ranging from 0.7 to 1.5 across multiple methods.
  arXiv  Detail & Related papers  (2025-12-12T00:08:03Z)
• TransNet: Transparent Object Manipulation Through Category-Level Pose Estimation [6.844391823478345]
  We propose a two-stage pipeline that estimates category-level transparent object pose using localized depth completion and surface normal estimation. Results show that TransNet achieves improved pose estimation accuracy on transparent objects. We use TransNet to build an autonomous transparent object manipulation system for robotic pick-and-place and pouring tasks.
  arXiv  Detail & Related papers  (2023-07-23T18:38:42Z)
• Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds [12.449232689517538]
  We propose a visual-tactile fusion framework for transparent object grasping. It includes grasping position detection, tactile calibration, and visual-tactile fusion based classification. The proposed framework synergizes the advantages of vision and touch, and greatly improves the grasping efficiency of transparent objects.
  arXiv  Detail & Related papers  (2022-11-30T02:27:41Z)
• TransNet: Category-Level Transparent Object Pose Estimation [6.844391823478345]
  The lack of distinguishing visual features makes transparent objects harder to detect and localize than opaque objects. A second challenge is that common depth sensors typically used for opaque object perception cannot obtain accurate depth measurements on transparent objects. We propose TransNet, a two-stage pipeline that learns to estimate category-level transparent object pose using localized depth completion and surface normal estimation.
  arXiv  Detail & Related papers  (2022-08-22T01:34:31Z)
• High-resolution Iterative Feedback Network for Camouflaged Object Detection [128.893782016078]
  Spotting camouflaged objects that are visually assimilated into the background is tricky for object detection algorithms. We aim to extract the high-resolution texture details to avoid the detail degradation that causes blurred vision in edges and boundaries. We introduce a novel HitNet to refine the low-resolution representations by high-resolution features in an iterative feedback manner.
  arXiv  Detail & Related papers  (2022-03-22T11:20:21Z)
• Generative Partial Visual-Tactile Fused Object Clustering [81.17645983141773]
  We propose a Generative Partial Visual-Tactile Fused (i.e., GPVTF) framework for object clustering. A conditional cross-modal clustering generative adversarial network is then developed to synthesize one modality conditioning on the other modality. To the end, two pseudo-label based KL-divergence losses are employed to update the corresponding modality-specific encoders.
  arXiv  Detail & Related papers  (2020-12-28T02:37:03Z)
• Refined Plane Segmentation for Cuboid-Shaped Objects by Leveraging Edge Detection [63.942632088208505]
  We propose a post-processing algorithm to align the segmented plane masks with edges detected in the image. This allows us to increase the accuracy of state-of-the-art approaches, while limiting ourselves to cuboid-shaped objects.
  arXiv  Detail & Related papers  (2020-03-28T18:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.