SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping

• URL: http://arxiv.org/abs/2505.24305v3
• Date: Fri, 20 Jun 2025 13:07:10 GMT
• Title: SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping
• Authors: Mingxu Zhang, Xiaoqi Li, Jiahui Xu, Kaichen Zhou, Hojin Bae, Yan Shen, Chuyan Xiong, Hao Dong,
• Abstract summary: We propose a training free framework SR3D that enables robotic grasping of transparent and specular objects from a single view observation.<n>Specifically, given single view RGB and depth images, SR3D first uses the external visual models to generate 3D reconstructed object mesh.<n>Then, the key idea is to determine the 3D object's pose and scale to accurately localize the reconstructed object back into its original depth corrupted 3D scene.
• Score: 7.222966501323922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in 3D robotic manipulation have improved grasping of everyday objects, but transparent and specular materials remain challenging due to depth sensing limitations. While several 3D reconstruction and depth completion approaches address these challenges, they suffer from setup complexity or limited observation information utilization. To address this, leveraging the power of single view 3D object reconstruction approaches, we propose a training free framework SR3D that enables robotic grasping of transparent and specular objects from a single view observation. Specifically, given single view RGB and depth images, SR3D first uses the external visual models to generate 3D reconstructed object mesh based on RGB image. Then, the key idea is to determine the 3D object's pose and scale to accurately localize the reconstructed object back into its original depth corrupted 3D scene. Therefore, we propose view matching and keypoint matching mechanisms,which leverage both the 2D and 3D's inherent semantic and geometric information in the observation to determine the object's 3D state within the scene, thereby reconstructing an accurate 3D depth map for effective grasp detection. Experiments in both simulation and real world show the reconstruction effectiveness of SR3D.

Related papers

• 3D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detection [58.78881632019072]
  We introduce the first end-to-end 3D Monocular Open-set Object Detector (3D-MOOD)<n>We lift the open-set 2D detection into 3D space through our designed 3D bounding box head.<n>We condition the object queries with geometry prior and overcome the generalization for 3D estimation across diverse scenes.
  arXiv  Detail & Related papers  (2025-07-31T13:56:41Z)
• DreamGrasp: Zero-Shot 3D Multi-Object Reconstruction from Partial-View Images for Robotic Manipulation [12.812918371506209]
  We introduce DreamGrasp, a framework that infers the unobserved parts of a scene.<n>By combining coarse 3D reconstruction, instance segmentation via contrastive learning, and text-guided instance-wise refinement, DreamGrasp circumvents limitations of prior methods.<n>Our experiments show that DreamGrasp not only recovers accurate object geometry but also supports downstream tasks like sequential decluttering and target retrieval with high success rates.
  arXiv  Detail & Related papers  (2025-07-08T03:12:49Z)
• Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images [66.77399370856462]
  Amodal3R is a conditional 3D generative model designed to reconstruct 3D objects from partial observations.<n>It learns to recover full 3D objects even in the presence of occlusions in real scenes.<n>It substantially outperforms existing methods that independently perform 2D amodal completion followed by 3D reconstruction.
  arXiv  Detail & Related papers  (2025-03-17T17:59:01Z)
• 3D Reconstruction of Objects in Hands without Real World 3D Supervision [12.70221786947807]
  We propose modules to leverage 3D supervision to scale up the learning of models for reconstructing hand-held objects. Specifically, we extract multiview 2D mask supervision from videos and 3D shape priors from shape collections. We use these indirect 3D cues to train occupancy networks that predict the 3D shape of objects from a single RGB image.
  arXiv  Detail & Related papers  (2023-05-04T17:56:48Z)
• MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices [78.20154723650333]
  High-quality 3D ground-truth shapes are critical for 3D object reconstruction evaluation. We introduce a novel multi-view RGBD dataset captured using a mobile device. We obtain precise 3D ground-truth shape without relying on high-end 3D scanners.
  arXiv  Detail & Related papers  (2023-03-03T14:02:50Z)
• 3D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimation [50.15926681475939]
  Inverse graphics aims to infer the 3D scene structure from 2D images. We introduce probabilistic modeling to quantify uncertainty and achieve robustness in 6D pose estimation tasks. 3DNEL effectively combines learned neural embeddings from RGB with depth information to improve robustness in sim-to-real 6D object pose estimation from RGB-D images.
  arXiv  Detail & Related papers  (2023-02-07T20:48:35Z)
• SSR-2D: Semantic 3D Scene Reconstruction from 2D Images [54.46126685716471]
  In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet.
  arXiv  Detail & Related papers  (2023-02-07T17:47:52Z)
• OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation [107.71752592196138]
  We propose OmniObject3D, a large vocabulary 3D object dataset with massive high-quality real-scanned 3D objects. It comprises 6,000 scanned objects in 190 daily categories, sharing common classes with popular 2D datasets. Each 3D object is captured with both 2D and 3D sensors, providing textured meshes, point clouds, multiview rendered images, and multiple real-captured videos.
  arXiv  Detail & Related papers  (2023-01-18T18:14:18Z)
• Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image [22.037472446683765]
  We learn a regular grid of 3D voxel features from the input image which is aligned with 3D scene space via a 3D feature lifting operator. Based on the 3D voxel features, our novel CenterNet-3D detection head formulates the 3D detection as keypoint detection in the 3D space. We devise an efficient coarse-to-fine reconstruction module, including coarse-level voxelization and a novel local PCA-SDF shape representation.
  arXiv  Detail & Related papers  (2021-11-04T18:30:37Z)
• Monocular Differentiable Rendering for Self-Supervised 3D Object Detection [21.825158925459732]
  3D object detection from monocular images is an ill-posed problem due to the projective entanglement of depth and scale. We present a novel self-supervised method for textured 3D shape reconstruction and pose estimation of rigid objects. Our method predicts the 3D location and meshes of each object in an image using differentiable rendering and a self-supervised objective.
  arXiv  Detail & Related papers  (2020-09-30T09:21:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.