Depth-aware Fusion Method based on Image and 4D Radar Spectrum for 3D Object Detection

• URL: http://arxiv.org/abs/2502.15516v1
• Date: Fri, 21 Feb 2025 15:14:30 GMT
• Title: Depth-aware Fusion Method based on Image and 4D Radar Spectrum for 3D Object Detection
• Authors: Yue Sun, Yeqiang Qian, Chunxiang Wang, Ming Yang,
• Abstract summary: 3D millimeter-wave radars can only provide range, Doppler, and azimuth information for objects.<n>This paper leverages these two highly complementary and cost-effective sensors, 4D millimeter-wave radar and camera.<n>We propose using GAN-based networks to generate depth images from radar spectra in the absence of depth sensors.
• Score: 32.96725310200148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safety and reliability are crucial for the public acceptance of autonomous driving. To ensure accurate and reliable environmental perception, intelligent vehicles must exhibit accuracy and robustness in various environments. Millimeter-wave radar, known for its high penetration capability, can operate effectively in adverse weather conditions such as rain, snow, and fog. Traditional 3D millimeter-wave radars can only provide range, Doppler, and azimuth information for objects. Although the recent emergence of 4D millimeter-wave radars has added elevation resolution, the radar point clouds remain sparse due to Constant False Alarm Rate (CFAR) operations. In contrast, cameras offer rich semantic details but are sensitive to lighting and weather conditions. Hence, this paper leverages these two highly complementary and cost-effective sensors, 4D millimeter-wave radar and camera. By integrating 4D radar spectra with depth-aware camera images and employing attention mechanisms, we fuse texture-rich images with depth-rich radar data in the Bird's Eye View (BEV) perspective, enhancing 3D object detection. Additionally, we propose using GAN-based networks to generate depth images from radar spectra in the absence of depth sensors, further improving detection accuracy.

Related papers

• TacoDepth: Towards Efficient Radar-Camera Depth Estimation with One-stage Fusion [54.46664104437454]
  We propose TacoDepth, an efficient and accurate Radar-Camera depth estimation model with one-stage fusion. Specifically, the graph-based Radar structure extractor and the pyramid-based Radar fusion module are designed. Compared with the previous state-of-the-art approach, TacoDepth improves depth accuracy and processing speed by 12.8% and 91.8%.
  arXiv  Detail & Related papers  (2025-04-16T05:25:04Z)
• RobuRCDet: Enhancing Robustness of Radar-Camera Fusion in Bird's Eye View for 3D Object Detection [68.99784784185019]
  Poor lighting or adverse weather conditions degrade camera performance.<n>Radar suffers from noise and positional ambiguity.<n>We propose RobuRCDet, a robust object detection model in BEV.
  arXiv  Detail & Related papers  (2025-02-18T17:17:38Z)
• TransRAD: Retentive Vision Transformer for Enhanced Radar Object Detection [6.163747364795787]
  We present TransRAD, a novel 3D radar object detection model. We propose Location-Aware NMS to mitigate the common issue of duplicate bounding boxes in deep radar object detection. Results demonstrate that TransRAD outperforms state-of-the-art methods in both 2D and 3D radar detection tasks.
  arXiv  Detail & Related papers  (2025-01-29T20:21:41Z)
• RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar [15.776076554141687]
  3D occupancy-based perception pipeline has significantly advanced autonomous driving. Current methods rely on LiDAR or camera inputs for 3D occupancy prediction. We introduce a novel approach that utilizes 4D imaging radar sensors for 3D occupancy prediction.
  arXiv  Detail & Related papers  (2024-05-22T21:48:17Z)
• Radar Fields: Frequency-Space Neural Scene Representations for FMCW Radar [62.51065633674272]
  We introduce Radar Fields - a neural scene reconstruction method designed for active radar imagers. Our approach unites an explicit, physics-informed sensor model with an implicit neural geometry and reflectance model to directly synthesize raw radar measurements. We validate the effectiveness of the method across diverse outdoor scenarios, including urban scenes with dense vehicles and infrastructure.
  arXiv  Detail & Related papers  (2024-05-07T20:44:48Z)
• MVFAN: Multi-View Feature Assisted Network for 4D Radar Object Detection [15.925365473140479]
  4D radar is recognized for its resilience and cost-effectiveness under adverse weather conditions. Unlike LiDAR and cameras, radar remains unimpaired by harsh weather conditions. We propose a framework for developing radar-based 3D object detection for autonomous vehicles.
  arXiv  Detail & Related papers  (2023-10-25T06:10:07Z)
• Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion [74.84019379368807]
  We propose a novel method named EchoFusion to skip the existing radar signal processing pipeline. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar to fuse with other sensors.
  arXiv  Detail & Related papers  (2023-07-31T09:53:50Z)
• SMURF: Spatial Multi-Representation Fusion for 3D Object Detection with 4D Imaging Radar [12.842457981088378]
  This paper introduces spatial multi-representation fusion (SMURF), a novel approach to 3D object detection using a single 4D imaging radar. SMURF mitigates measurement inaccuracy caused by limited angular resolution and multi-path propagation of radar signals. Experimental evaluations on View-of-Delft (VoD) and TJ4DRadSet datasets demonstrate the effectiveness and generalization ability of SMURF.
  arXiv  Detail & Related papers  (2023-07-20T11:33:46Z)
• CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection [12.557361522985898]
  We propose a camera-radar matching network CramNet to fuse the sensor readings from camera and radar in a joint 3D space. Our method supports training with sensor modality dropout, which leads to robust 3D object detection, even when a camera or radar sensor suddenly malfunctions on a vehicle.
  arXiv  Detail & Related papers  (2022-10-17T17:18:47Z)
• R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes [69.6715406227469]
  Self-supervised monocular depth estimation in driving scenarios has achieved comparable performance to supervised approaches. We present R4Dyn, a novel set of techniques to use cost-efficient radar data on top of a self-supervised depth estimation framework.
  arXiv  Detail & Related papers  (2021-08-10T17:57:03Z)
• RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects [73.80316195652493]
  We tackle the problem of exploiting Radar for perception in the context of self-driving cars. We propose a new solution that exploits both LiDAR and Radar sensors for perception. Our approach, dubbed RadarNet, features a voxel-based early fusion and an attention-based late fusion.
  arXiv  Detail & Related papers  (2020-07-28T17:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.