Frustum Fusion: Pseudo-LiDAR and LiDAR Fusion for 3D Detection

• URL: http://arxiv.org/abs/2111.04780v1
• Date: Mon, 8 Nov 2021 19:29:59 GMT
• Title: Frustum Fusion: Pseudo-LiDAR and LiDAR Fusion for 3D Detection
• Authors: Farzin Negahbani, Onur Berk T\"ore, Fatma G\"uney and Baris Akgun
• Abstract summary: We propose a novel data fusion algorithm to combine accurate point clouds with dense but less accurate point clouds obtained from stereo pairs. We train multiple 3D object detection methods and show that our fusion strategy consistently improves the performance of detectors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Most autonomous vehicles are equipped with LiDAR sensors and stereo cameras. The former is very accurate but generates sparse data, whereas the latter is dense, has rich texture and color information but difficult to extract robust 3D representations from. In this paper, we propose a novel data fusion algorithm to combine accurate point clouds with dense but less accurate point clouds obtained from stereo pairs. We develop a framework to integrate this algorithm into various 3D object detection methods. Our framework starts with 2D detections from both of the RGB images, calculates frustums and their intersection, creates Pseudo-LiDAR data from the stereo images, and fills in the parts of the intersection region where the LiDAR data is lacking with the dense Pseudo-LiDAR points. We train multiple 3D object detection methods and show that our fusion strategy consistently improves the performance of detectors.

Related papers

• Fully Sparse Fusion for 3D Object Detection [69.32694845027927]
  Currently prevalent multimodal 3D detection methods are built upon LiDAR-based detectors that usually use dense Bird's-Eye-View feature maps. Fully sparse architecture is gaining attention as they are highly efficient in long-range perception. In this paper, we study how to effectively leverage image modality in the emerging fully sparse architecture.
  arXiv  Detail & Related papers  (2023-04-24T17:57:43Z)
• Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection [11.575945934519442]
  LiDAR and camera fusion techniques are promising for achieving 3D object detection in autonomous driving. Most multi-modal 3D object detection frameworks integrate semantic knowledge from 2D images into 3D LiDAR point clouds. We propose a general multi-modal fusion framework Multi-Sem Fusion (MSF) to fuse the semantic information from both the 2D image and 3D points scene parsing results.
  arXiv  Detail & Related papers  (2022-12-10T10:54:41Z)
• MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer for Autonomous Driving [0.0]
  We propose MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer architecture to fuse image and LiDAR features to improve the detection accuracy. Our end-to-end single-stage, anchor-free and NMS-free network takes in multi-view images and LiDAR point clouds and predicts 3D bounding boxes. MSF3DDETR network is trained end-to-end on the nuScenes dataset using Hungarian algorithm based bipartite matching and set-to-set loss inspired by DETR.
  arXiv  Detail & Related papers  (2022-10-27T10:55:15Z)
• Bridged Transformer for Vision and Point Cloud 3D Object Detection [92.86856146086316]
  Bridged Transformer (BrT) is an end-to-end architecture for 3D object detection. BrT learns to identify 3D and 2D object bounding boxes from both points and image patches. We experimentally show that BrT surpasses state-of-the-art methods on SUN RGB-D and ScanNetV2 datasets.
  arXiv  Detail & Related papers  (2022-10-04T05:44:22Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• Dense Voxel Fusion for 3D Object Detection [10.717415797194896]
  Voxel Fusion (DVF) is a sequential fusion method that generates multi-scale dense voxel feature representations. We train directly with ground truth 2D bounding box labels, avoiding noisy, detector-specific, 2D predictions. We show that our proposed multi-modal training strategy results in better generalization compared to training using erroneous 2D predictions.
  arXiv  Detail & Related papers  (2022-03-02T04:51:31Z)
• VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion [62.24001258298076]
  VPFNet is a new architecture that cleverly aligns and aggregates the point cloud and image data at the virtual' points. Our VPFNet achieves 83.21% moderate 3D AP and 91.86% moderate BEV AP on the KITTI test set, ranking the 1st since May 21th, 2021.
  arXiv  Detail & Related papers  (2021-11-29T08:51:20Z)
• Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation [81.08111209632501]
  We propose a geometry-aware stereo-LiDAR fusion network for long-range depth estimation. We exploit sparse and accurate point clouds as a cue for guiding correspondences of stereo images in a unified 3D volume space. Our network achieves state-of-the-art performance on the KITTI and the Virtual- KITTI datasets.
  arXiv  Detail & Related papers  (2021-03-24T03:24:46Z)
• RoIFusion: 3D Object Detection from LiDAR and Vision [7.878027048763662]
  We propose a novel fusion algorithm by projecting a set of 3D Region of Interests (RoIs) from the point clouds to the 2D RoIs of the corresponding the images. Our approach achieves state-of-the-art performance on the KITTI 3D object detection challenging benchmark.
  arXiv  Detail & Related papers  (2020-09-09T20:23:27Z)
• Cross-Modality 3D Object Detection [63.29935886648709]
  We present a novel two-stage multi-modal fusion network for 3D object detection. The whole architecture facilitates two-stage fusion. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.
  arXiv  Detail & Related papers  (2020-08-16T11:01:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.