Sparse-to-Dense LiDAR Point Generation by LiDAR-Camera Fusion for 3D Object Detection

• URL: http://arxiv.org/abs/2409.14985v2
• Date: Tue, 24 Sep 2024 16:20:30 GMT
• Title: Sparse-to-Dense LiDAR Point Generation by LiDAR-Camera Fusion for 3D Object Detection
• Authors: Minseung Lee, Seokha Moon, Seung Joon Lee, Jinkyu Kim,
• Abstract summary: We propose the LiDAR-Camera Augmentation Network (LCANet), a novel framework that reconstructs LiDAR point cloud data by fusing 2D image features. LCANet fuses data from LiDAR sensors by projecting image features into the 3D space, integrating semantic information into the point cloud data. This fusion effectively compensates for LiDAR's weakness in detecting objects at long distances, which are often represented by sparse points.
• Score: 9.076003184833557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurately detecting objects at long distances remains a critical challenge in 3D object detection when relying solely on LiDAR sensors due to the inherent limitations of data sparsity. To address this issue, we propose the LiDAR-Camera Augmentation Network (LCANet), a novel framework that reconstructs LiDAR point cloud data by fusing 2D image features, which contain rich semantic information, generating additional points to improve detection accuracy. LCANet fuses data from LiDAR sensors and cameras by projecting image features into the 3D space, integrating semantic information into the point cloud data. This fused data is then encoded to produce 3D features that contain both semantic and spatial information, which are further refined to reconstruct final points before bounding box prediction. This fusion effectively compensates for LiDAR's weakness in detecting objects at long distances, which are often represented by sparse points. Additionally, due to the sparsity of many objects in the original dataset, which makes effective supervision for point generation challenging, we employ a point cloud completion network to create a complete point cloud dataset that supervises the generation of dense point clouds in our network. Extensive experiments on the KITTI and Waymo datasets demonstrate that LCANet significantly outperforms existing models, particularly in detecting sparse and distant objects.

Related papers

• VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection [80.62052650370416]
  monocular 3D object detection holds significant importance across various applications, including autonomous driving and robotics. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations.
  arXiv  Detail & Related papers  (2024-04-15T03:12:12Z)
• VirtualPainting: Addressing Sparsity with Virtual Points and Distance-Aware Data Augmentation for 3D Object Detection [3.5259183508202976]
  We present an innovative approach that involves the generation of virtual LiDAR points using camera images. We also enhance these virtual points with semantic labels obtained from image-based segmentation networks. Our approach offers a versatile solution that can be seamlessly integrated into various 3D frameworks and 2D semantic segmentation methods.
  arXiv  Detail & Related papers  (2023-12-26T18:03:05Z)
• Semantics-aware LiDAR-Only Pseudo Point Cloud Generation for 3D Object Detection [0.7234862895932991]
  Recent advances introduced pseudo-LiDAR, i.e., synthetic dense point clouds, using additional modalities such as cameras to enhance 3D object detection. We present a novel LiDAR-only framework that augments raw scans with dense pseudo point clouds by relying on LiDAR sensors and scene semantics.
  arXiv  Detail & Related papers  (2023-09-16T09:18:47Z)
• AGO-Net: Association-Guided 3D Point Cloud Object Detection Network [86.10213302724085]
  We propose a novel 3D detection framework that associates intact features for objects via domain adaptation. We achieve new state-of-the-art performance on the KITTI 3D detection benchmark in both accuracy and speed.
  arXiv  Detail & Related papers  (2022-08-24T16:54:38Z)
• VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion [62.24001258298076]
  VPFNet is a new architecture that cleverly aligns and aggregates the point cloud and image data at the virtual' points. Our VPFNet achieves 83.21% moderate 3D AP and 91.86% moderate BEV AP on the KITTI test set, ranking the 1st since May 21th, 2021.
  arXiv  Detail & Related papers  (2021-11-29T08:51:20Z)
• Anchor-free 3D Single Stage Detector with Mask-Guided Attention for Point Cloud [79.39041453836793]
  We develop a novel single-stage 3D detector for point clouds in an anchor-free manner. We overcome this by converting the voxel-based sparse 3D feature volumes into the sparse 2D feature maps. We propose an IoU-based detection confidence re-calibration scheme to improve the correlation between the detection confidence score and the accuracy of the bounding box regression.
  arXiv  Detail & Related papers  (2021-08-08T13:42:13Z)
• PC-DAN: Point Cloud based Deep Affinity Network for 3D Multi-Object Tracking (Accepted as an extended abstract in JRDB-ACT Workshop at CVPR21) [68.12101204123422]
  A point cloud is a dense compilation of spatial data in 3D coordinates. We propose a PointNet-based approach for 3D Multi-Object Tracking (MOT)
  arXiv  Detail & Related papers  (2021-06-03T05:36:39Z)
• SIENet: Spatial Information Enhancement Network for 3D Object Detection from Point Cloud [20.84329063509459]
  LiDAR-based 3D object detection pushes forward an immense influence on autonomous vehicles. Due to the limitation of the intrinsic properties of LiDAR, fewer points are collected at the objects farther away from the sensor. To address the challenge, we propose a novel two-stage 3D object detection framework, named SIENet.
  arXiv  Detail & Related papers  (2021-03-29T07:45:09Z)
• RoIFusion: 3D Object Detection from LiDAR and Vision [7.878027048763662]
  We propose a novel fusion algorithm by projecting a set of 3D Region of Interests (RoIs) from the point clouds to the 2D RoIs of the corresponding the images. Our approach achieves state-of-the-art performance on the KITTI 3D object detection challenging benchmark.
  arXiv  Detail & Related papers  (2020-09-09T20:23:27Z)
• Cross-Modality 3D Object Detection [63.29935886648709]
  We present a novel two-stage multi-modal fusion network for 3D object detection. The whole architecture facilitates two-stage fusion. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.
  arXiv  Detail & Related papers  (2020-08-16T11:01:20Z)
• 3D Object Detection From LiDAR Data Using Distance Dependent Feature Extraction [7.04185696830272]
  This work proposes an improvement for 3D object detectors by taking into account the properties of LiDAR point clouds over distance. Results show that training separate networks for close-range and long-range objects boosts performance for all KITTI benchmark difficulties.
  arXiv  Detail & Related papers  (2020-03-02T13:16:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.