A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection

• URL: http://arxiv.org/abs/2005.07431v1
• Date: Fri, 15 May 2020 09:28:01 GMT
• Title: A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection
• Authors: Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz and Markus Lienkamp
• Abstract summary: This research aims to enhance current 2D object detection networks by fusing camera data and projected sparse radar data in the network layers. The proposed CameraRadarFusionNet (CRF-Net) automatically learns at which level the fusion of the sensor data is most beneficial for the detection result. BlackIn, a training strategy inspired by Dropout, focuses the learning on a specific sensor type.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object detection in camera images, using deep learning has been proven successfully in recent years. Rising detection rates and computationally efficient network structures are pushing this technique towards application in production vehicles. Nevertheless, the sensor quality of the camera is limited in severe weather conditions and through increased sensor noise in sparsely lit areas and at night. Our approach enhances current 2D object detection networks by fusing camera data and projected sparse radar data in the network layers. The proposed CameraRadarFusionNet (CRF-Net) automatically learns at which level the fusion of the sensor data is most beneficial for the detection result. Additionally, we introduce BlackIn, a training strategy inspired by Dropout, which focuses the learning on a specific sensor type. We show that the fusion network is able to outperform a state-of-the-art image-only network for two different datasets. The code for this research will be made available to the public at: https://github.com/TUMFTM/CameraRadarFusionNet.

Related papers

• Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor [58.305341034419136]
  We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
  arXiv  Detail & Related papers  (2023-08-28T07:56:13Z)
• Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion [74.84019379368807]
  We propose a novel method named EchoFusion to skip the existing radar signal processing pipeline. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar to fuse with other sensors.
  arXiv  Detail & Related papers  (2023-07-31T09:53:50Z)
• HVDetFusion: A Simple and Robust Camera-Radar Fusion Framework [10.931114142452895]
  Current SOTA algorithm combines Camera and Lidar sensors, limited by the high price of Lidar. HVDetFusion is a multi-modal detection algorithm that supports pure camera data as input for detection, but also can perform fusion input of radar data and camera data. HVDetFusion achieves the new state-of-the-art 67.4% NDS on the challenging nuScenes test set among all camera-radar 3D object detectors.
  arXiv  Detail & Related papers  (2023-07-21T03:08:28Z)
• DensePose From WiFi [86.61881052177228]
  We develop a deep neural network that maps the phase and amplitude of WiFi signals to UV coordinates within 24 human regions. Our model can estimate the dense pose of multiple subjects, with comparable performance to image-based approaches.
  arXiv  Detail & Related papers  (2022-12-31T16:48:43Z)
• CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection [12.557361522985898]
  We propose a camera-radar matching network CramNet to fuse the sensor readings from camera and radar in a joint 3D space. Our method supports training with sensor modality dropout, which leads to robust 3D object detection, even when a camera or radar sensor suddenly malfunctions on a vehicle.
  arXiv  Detail & Related papers  (2022-10-17T17:18:47Z)
• Drone Detection and Tracking in Real-Time by Fusion of Different Sensing Modalities [66.4525391417921]
  We design and evaluate a multi-sensor drone detection system. Our solution integrates a fish-eye camera as well to monitor a wider part of the sky and steer the other cameras towards objects of interest. The thermal camera is shown to be a feasible solution as good as the video camera, even if the camera employed here has a lower resolution.
  arXiv  Detail & Related papers  (2022-07-05T10:00:58Z)
• Radar Voxel Fusion for 3D Object Detection [0.0]
  This paper develops a low-level sensor fusion network for 3D object detection. The radar sensor fusion proves especially beneficial in inclement conditions such as rain and night scenes.
  arXiv  Detail & Related papers  (2021-06-26T20:34:12Z)
• Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion [56.85837052421469]
  Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion.
  arXiv  Detail & Related papers  (2021-03-30T21:22:26Z)
• Multi-Modal Depth Estimation Using Convolutional Neural Networks [0.8701566919381223]
  This paper addresses the problem of dense depth predictions from sparse distance sensor data and a single camera image on challenging weather conditions. It explores the significance of different sensor modalities such as camera, Radar, and Lidar for estimating depth by applying Deep Learning approaches.
  arXiv  Detail & Related papers  (2020-12-17T15:31:49Z)
• YOdar: Uncertainty-based Sensor Fusion for Vehicle Detection with Camera and Radar Sensors [4.396860522241306]
  We present an uncertainty-based method for sensor fusion with camera and radar data. In our experiments we combine the YOLOv3 object detection network with a customized $1D$ radar segmentation network. Our experiments show, that this approach of uncertainty aware fusion significantly gains performance compared to single sensor baselines.
  arXiv  Detail & Related papers  (2020-10-07T10:40:02Z)
• RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects [73.80316195652493]
  We tackle the problem of exploiting Radar for perception in the context of self-driving cars. We propose a new solution that exploits both LiDAR and Radar sensors for perception. Our approach, dubbed RadarNet, features a voxel-based early fusion and an attention-based late fusion.
  arXiv  Detail & Related papers  (2020-07-28T17:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.