Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

• URL: http://arxiv.org/abs/2004.05224v2
• Date: Wed, 9 Sep 2020 14:12:13 GMT
• Title: Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• Authors: Yaodong Cui, Ren Chen, Wenbo Chu, Long Chen, Daxin Tian, Ying Li, Dongpu Cao
• Abstract summary: Camera-LiDAR fusion is becoming an emerging research theme. This paper reviews recent deep-learning-based data fusion approaches that leverage both image and point cloud. We identify gaps and over-looked challenges between current academic researches and real-world applications.
• Score: 15.10767676137607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this paper devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.

Related papers

• Addressing Data Misalignment in Image-LiDAR Fusion on Point Cloud Segmentation [0.391609470658968]
  We show that the projected positions of LiDAR points often misalign on the corresponding image. In this paper, we would like to address this problem carefully, with a specific focus on the nuScenes dataset and the SOTA of fusion models 2DPASS.
  arXiv  Detail & Related papers  (2023-09-26T13:41:30Z)
• Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review [7.835577409160127]
  This review focuses on perception tasks related to object detection and semantic segmentation. In the review, we address interrogative questions, including "why to fuse", "what to fuse", "where to fuse", "when to fuse", and "how to fuse"
  arXiv  Detail & Related papers  (2023-04-20T15:48:50Z)
• Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics [77.34726150561087]
  This work aims to carry out a study on the current scenario of camera and radar-based perception for ADAS and autonomous vehicles. Concepts and characteristics related to both sensors, as well as to their fusion, are presented. We give an overview of the Deep Learning-based detection and segmentation tasks, and the main datasets, metrics, challenges, and open questions in vehicle perception.
  arXiv  Detail & Related papers  (2023-03-08T00:48:32Z)
• Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving [58.879758550901364]
  Recent perception systems enhance spatial understanding with sensor fusion but often lack full environmental context. We introduce a framework that integrates three cameras to emulate the human field of view, coupled with top-down bird-eye-view semantic data to enhance contextual representation. Our method achieves displacement error by 0.67m in open-loop settings, surpassing current methods by 6.9% on the nuScenes dataset.
  arXiv  Detail & Related papers  (2022-10-13T05:56:20Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation [101.55622133406446]
  We propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:58:47Z)
• LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation [78.74202673902303]
  We propose a coarse-tofine LiDAR and camera fusion-based network (termed as LIF-Seg) for LiDAR segmentation. The proposed method fully utilizes the contextual information of images and introduces a simple but effective early-fusion strategy. The cooperation of these two components leads to the success of the effective camera-LiDAR fusion.
  arXiv  Detail & Related papers  (2021-08-17T08:53:11Z)
• MmWave Radar and Vision Fusion based Object Detection for Autonomous Driving: A Survey [15.316597644398188]
  Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods.
  arXiv  Detail & Related papers  (2021-08-06T08:38:42Z)
• Multi-Modal 3D Object Detection in Autonomous Driving: a Survey [10.913958563906931]
  Self-driving cars are equipped with a suite of sensors to conduct robust and accurate environment perception. As the number and type of sensors keep increasing, combining them for better perception is becoming a natural trend. This survey devotes to review recent fusion-based 3D detection deep learning models that leverage multiple sensor data sources.
  arXiv  Detail & Related papers  (2021-06-24T02:52:12Z)
• Lighting the Darkness in the Deep Learning Era [118.35081853500411]
  Low-light image enhancement (LLIE) aims at improving the perception or interpretability of an image captured in an environment with poor illumination. Recent advances in this area are dominated by deep learning-based solutions. We provide a comprehensive survey to cover various aspects ranging from algorithm taxonomy to unsolved open issues.
  arXiv  Detail & Related papers  (2021-04-21T19:12:19Z)
• 3D Point Cloud Processing and Learning for Autonomous Driving [26.285659927609213]
  We present a review of 3D point cloud processing and learning for autonomous driving. LiDAR sensors collect 3D point clouds that precisely record the external surfaces of objects and scenes.
  arXiv  Detail & Related papers  (2020-03-01T22:13:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.