GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects

• URL: http://arxiv.org/abs/2503.17106v1
• Date: Fri, 21 Mar 2025 12:46:38 GMT
• Title: GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objects
• Authors: Yizhe Liu, Tong Jia, Da Cai, Hao Wang, Dongyue Chen,
• Abstract summary: We propose a geometry-aware assisted depth completion method for transparent and specular objects.<n>Specifically, we back-project the input depth to a point cloud and build the 3D branch to extract hierarchical scene-level 3D structural features.<n>Our method significantly enhances the performance of downstream robotic grasping tasks.
• Score: 5.227530976208222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transparent and specular objects are frequently encountered in daily life, factories, and laboratories. However, due to the unique optical properties, the depth information on these objects is usually incomplete and inaccurate, which poses significant challenges for downstream robotics tasks. Therefore, it is crucial to accurately restore the depth information of transparent and specular objects. Previous depth completion methods for these objects usually use RGB information as an additional channel of the depth image to perform depth prediction. Due to the poor-texture characteristics of transparent and specular objects, these methods that rely heavily on color information tend to generate structure-less depth predictions. Moreover, these 2D methods cannot effectively explore the 3D structure hidden in the depth channel, resulting in depth ambiguity. To this end, we propose a geometry-aware assisted depth completion method for transparent and specular objects, which focuses on exploring the 3D structural cues of the scene. Specifically, besides extracting 2D features from RGB-D input, we back-project the input depth to a point cloud and build the 3D branch to extract hierarchical scene-level 3D structural features. To exploit 3D geometric information, we design several gated cross-modal fusion modules to effectively propagate multi-level 3D geometric features to the image branch. In addition, we propose an adaptive correlation aggregation strategy to appropriately assign 3D features to the corresponding 2D features. Extensive experiments on ClearGrasp, OOD, TransCG, and STD datasets show that our method outperforms other state-of-the-art methods. We further demonstrate that our method significantly enhances the performance of downstream robotic grasping tasks.

Related papers

• GAC-Net_Geometric and attention-based Network for Depth Completion [10.64600095082433]
  This paper proposes a depth completion network combining channel attention mechanism and 3D global feature perception (CGA-Net)<n> Experiments on the KITTI depth completion dataset show that CGA-Net can significantly improve the prediction accuracy of dense depth maps.
  arXiv  Detail & Related papers  (2025-01-14T10:24:20Z)
• OPEN: Object-wise Position Embedding for Multi-view 3D Object Detection [102.0744303467713]
  We propose a new multi-view 3D object detector named OPEN. Our main idea is to effectively inject object-wise depth information into the network through our proposed object-wise position embedding. OPEN achieves a new state-of-the-art performance with 64.4% NDS and 56.7% mAP on the nuScenes test benchmark.
  arXiv  Detail & Related papers  (2024-07-15T14:29:15Z)
• Perspective-aware Convolution for Monocular 3D Object Detection [2.33877878310217]
  We propose a novel perspective-aware convolutional layer that captures long-range dependencies in images. By enforcing convolutional kernels to extract features along the depth axis of every image pixel, we incorporates perspective information into network architecture. We demonstrate improved performance on the KITTI3D dataset, achieving a 23.9% average precision in the easy benchmark.
  arXiv  Detail & Related papers  (2023-08-24T17:25:36Z)
• MonoPGC: Monocular 3D Object Detection with Pixel Geometry Contexts [6.639648061168067]
  We propose MonoPGC, a novel end-to-end Monocular 3D object detection framework with rich Pixel Geometry Contexts. We introduce the pixel depth estimation as our auxiliary task and design depth cross-attention pyramid module (DCPM) to inject local and global depth geometry knowledge into visual features. In addition, we present the depth-space-aware transformer (DSAT) to integrate 3D space position and depth-aware features efficiently.
  arXiv  Detail & Related papers  (2023-02-21T09:21:58Z)
• OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection [51.153003057515754]
  OPA-3D is a single-stage, end-to-end, Occlusion-Aware Pixel-Wise Aggregation network. It jointly estimates dense scene depth with depth-bounding box residuals and object bounding boxes. It outperforms state-of-the-art methods on the main Car category.
  arXiv  Detail & Related papers  (2022-11-02T14:19:13Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection [70.71934539556916]
  We learn geometry-guided depth estimation with projective modeling to advance monocular 3D object detection. Specifically, a principled geometry formula with projective modeling of 2D and 3D depth predictions in the monocular 3D object detection network is devised. Our method remarkably improves the detection performance of the state-of-the-art monocular-based method without extra data by 2.80% on the moderate test setting.
  arXiv  Detail & Related papers  (2021-07-29T12:30:39Z)
• VR3Dense: Voxel Representation Learning for 3D Object Detection and Monocular Dense Depth Reconstruction [0.951828574518325]
  We introduce a method for jointly training 3D object detection and monocular dense depth reconstruction neural networks. It takes as inputs, a LiDAR point-cloud, and a single RGB image during inference and produces object pose predictions as well as a densely reconstructed depth map. While our object detection is trained in a supervised manner, the depth prediction network is trained with both self-supervised and supervised loss functions.
  arXiv  Detail & Related papers  (2021-04-13T04:25:54Z)
• Learning Joint 2D-3D Representations for Depth Completion [90.62843376586216]
  We design a simple yet effective neural network block that learns to extract joint 2D and 3D features. Specifically, the block consists of two domain-specific sub-networks that apply 2D convolution on image pixels and continuous convolution on 3D points.
  arXiv  Detail & Related papers  (2020-12-22T22:58:29Z)
• DSGN: Deep Stereo Geometry Network for 3D Object Detection [79.16397166985706]
  There is a large performance gap between image-based and LiDAR-based 3D object detectors. Our method, called Deep Stereo Geometry Network (DSGN), significantly reduces this gap. For the first time, we provide a simple and effective one-stage stereo-based 3D detection pipeline.
  arXiv  Detail & Related papers  (2020-01-10T11:44:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.