VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection

• URL: http://arxiv.org/abs/2404.00149v1
• Date: Fri, 29 Mar 2024 20:43:55 GMT
• Title: VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection
• Authors: Zihua Liu, Hiroki Sakuma, Masatoshi Okutomi,
• Abstract summary: Monocular 3D object detection poses a significant challenge in 3D scene understanding. Existing methods heavily rely on supervised learning using abundant 3D labels. We propose a novel weakly supervised 3D object detection framework named VSRD.
• Score: 11.061100776969383
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular 3D object detection poses a significant challenge in 3D scene understanding due to its inherently ill-posed nature in monocular depth estimation. Existing methods heavily rely on supervised learning using abundant 3D labels, typically obtained through expensive and labor-intensive annotation on LiDAR point clouds. To tackle this problem, we propose a novel weakly supervised 3D object detection framework named VSRD (Volumetric Silhouette Rendering for Detection) to train 3D object detectors without any 3D supervision but only weak 2D supervision. VSRD consists of multi-view 3D auto-labeling and subsequent training of monocular 3D object detectors using the pseudo labels generated in the auto-labeling stage. In the auto-labeling stage, we represent the surface of each instance as a signed distance field (SDF) and render its silhouette as an instance mask through our proposed instance-aware volumetric silhouette rendering. To directly optimize the 3D bounding boxes through rendering, we decompose the SDF of each instance into the SDF of a cuboid and the residual distance field (RDF) that represents the residual from the cuboid. This mechanism enables us to optimize the 3D bounding boxes in an end-to-end manner by comparing the rendered instance masks with the ground truth instance masks. The optimized 3D bounding boxes serve as effective training data for 3D object detection. We conduct extensive experiments on the KITTI-360 dataset, demonstrating that our method outperforms the existing weakly supervised 3D object detection methods. The code is available at https://github.com/skmhrk1209/VSRD.

Related papers

• Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data [57.53523870705433]
  We propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det. OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. It employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors.
  arXiv  Detail & Related papers  (2024-11-23T21:37:21Z)
• Weakly Supervised 3D Object Detection via Multi-Level Visual Guidance [72.6809373191638]
  We propose a framework to study how to leverage constraints between 2D and 3D domains without requiring any 3D labels. Specifically, we design a feature-level constraint to align LiDAR and image features based on object-aware regions. Second, the output-level constraint is developed to enforce the overlap between 2D and projected 3D box estimations. Third, the training-level constraint is utilized by producing accurate and consistent 3D pseudo-labels that align with the visual data.
  arXiv  Detail & Related papers  (2023-12-12T18:57:25Z)
• CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection [57.44434974289945]
  We propose Contextualized Multi-Stage Refinement for 3D Object Detection (CMR3D) framework. Our framework takes a 3D scene as input and strives to explicitly integrate useful contextual information of the scene. In addition to 3D object detection, we investigate the effectiveness of our framework for the problem of 3D object counting.
  arXiv  Detail & Related papers  (2022-09-13T05:26:09Z)
• Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement [93.77156425817178]
  We propose a weakly-supervised approach for 3D object detection, which makes it possible to train strong 3D detector with position-level annotations. Our method, namely Back to Reality (BR), makes use of synthetic 3D shapes to convert the weak labels into fully-annotated virtual scenes. With less than 5% of the labeling labor, we achieve comparable detection performance with some popular fully-supervised approaches on the widely used ScanNet dataset.
  arXiv  Detail & Related papers  (2022-03-10T08:51:32Z)
• FGR: Frustum-Aware Geometric Reasoning for Weakly Supervised 3D Vehicle Detection [81.79171905308827]
  We propose frustum-aware geometric reasoning (FGR) to detect vehicles in point clouds without any 3D annotations. Our method consists of two stages: coarse 3D segmentation and 3D bounding box estimation. It is able to accurately detect objects in 3D space with only 2D bounding boxes and sparse point clouds.
  arXiv  Detail & Related papers  (2021-05-17T07:29:55Z)
• Learning Monocular 3D Vehicle Detection without 3D Bounding Box Labels [0.09558392439655011]
  Training of 3D object detectors requires datasets with 3D bounding box labels for supervision that have to be generated by hand-labeling. We propose a network architecture and training procedure for learning monocular 3D object detection without 3D bounding box labels. We evaluate the proposed algorithm on the real-world KITTI dataset and achieve promising performance in comparison to state-of-the-art methods requiring 3D bounding box labels for training.
  arXiv  Detail & Related papers  (2020-10-07T16:24:46Z)
• Monocular Differentiable Rendering for Self-Supervised 3D Object Detection [21.825158925459732]
  3D object detection from monocular images is an ill-posed problem due to the projective entanglement of depth and scale. We present a novel self-supervised method for textured 3D shape reconstruction and pose estimation of rigid objects. Our method predicts the 3D location and meshes of each object in an image using differentiable rendering and a self-supervised objective.
  arXiv  Detail & Related papers  (2020-09-30T09:21:43Z)
• Weakly Supervised 3D Object Detection from Point Clouds [27.70180601788613]
  3D object detection aims to detect and localize the 3D bounding boxes of objects belonging to specific classes. Existing 3D object detectors rely on annotated 3D bounding boxes during training, while these annotations could be expensive to obtain and only accessible in limited scenarios. We propose VS3D, a framework for weakly supervised 3D object detection from point clouds without using any ground truth 3D bounding box for training.
  arXiv  Detail & Related papers  (2020-07-28T03:30:11Z)
• DSGN: Deep Stereo Geometry Network for 3D Object Detection [79.16397166985706]
  There is a large performance gap between image-based and LiDAR-based 3D object detectors. Our method, called Deep Stereo Geometry Network (DSGN), significantly reduces this gap. For the first time, we provide a simple and effective one-stage stereo-based 3D detection pipeline.
  arXiv  Detail & Related papers  (2020-01-10T11:44:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.