CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementarity

• URL: http://arxiv.org/abs/2310.02815v3
• Date: Sun, 15 Sep 2024 15:11:55 GMT
• Title: CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementarity
• Authors: Hao Shi, Chengshan Pang, Jiaming Zhang, Kailun Yang, Yuhao Wu, Huajian Ni, Yining Lin, Rainer Stiefelhagen, Kaiwei Wang,
• Abstract summary: We develop Complementary-BEV, a novel end-to-end monocular 3D object detection framework. We conduct extensive experiments on the public 3D detection benchmarks of roadside camera-based DAIR-V2X-I and Rope3D. For the first time, the vehicle AP score of a camera model reaches 80% on DAIR-V2X-I in terms of easy mode.
• Score: 34.025530326420146
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Roadside camera-driven 3D object detection is a crucial task in intelligent transportation systems, which extends the perception range beyond the limitations of vision-centric vehicles and enhances road safety. While previous studies have limitations in using only depth or height information, we find both depth and height matter and they are in fact complementary. The depth feature encompasses precise geometric cues, whereas the height feature is primarily focused on distinguishing between various categories of height intervals, essentially providing semantic context. This insight motivates the development of Complementary-BEV (CoBEV), a novel end-to-end monocular 3D object detection framework that integrates depth and height to construct robust BEV representations. In essence, CoBEV estimates each pixel's depth and height distribution and lifts the camera features into 3D space for lateral fusion using the newly proposed two-stage complementary feature selection (CFS) module. A BEV feature distillation framework is also seamlessly integrated to further enhance the detection accuracy from the prior knowledge of the fusion-modal CoBEV teacher. We conduct extensive experiments on the public 3D detection benchmarks of roadside camera-based DAIR-V2X-I and Rope3D, as well as the private Supremind-Road dataset, demonstrating that CoBEV not only achieves the accuracy of the new state-of-the-art, but also significantly advances the robustness of previous methods in challenging long-distance scenarios and noisy camera disturbance, and enhances generalization by a large margin in heterologous settings with drastic changes in scene and camera parameters. For the first time, the vehicle AP score of a camera model reaches 80% on DAIR-V2X-I in terms of easy mode. The source code will be made publicly available at https://github.com/MasterHow/CoBEV.

Related papers

• ROA-BEV: 2D Region-Oriented Attention for BEV-based 3D Object [14.219472370221029]
  We propose 2D Region-oriented Attention for a BEV-based 3D Object Detection Network (ROA-BEV) Our method increases the information content of ROA through a multi-scale structure. Experiments on nuScenes show that ROA-BEV improves the performance based on BEVDet and BEVDepth.
  arXiv  Detail & Related papers  (2024-10-14T08:51:56Z)
• HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective [11.841338298700421]
  We propose a novel 3D object detection framework integrating Spatial Former and Voxel Pooling Former to enhance 2D-to-3D projection based on height estimation. Experiments were conducted using the Rope3D and DAIR-V2X-I dataset, and the results demonstrated the outperformance of the proposed algorithm in the detection of both vehicles and cyclists.
  arXiv  Detail & Related papers  (2024-10-10T09:37:33Z)
• BEVSpread: Spread Voxel Pooling for Bird's-Eye-View Representation in Vision-based Roadside 3D Object Detection [47.74067616658986]
  Vision-based roadside 3D object detection has attracted rising attention in autonomous driving domain. Inspired by this insight, we propose a novel voxel pooling strategy to reduce such error, dubbed BEVSpread. BeVSpread can significantly improve the performance of existing frustum-based BEV methods by a large margin.
  arXiv  Detail & Related papers  (2024-06-13T03:33:36Z)
• Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception [5.920946963698008]
  We introduce HyDRa, a novel camera-radar fusion architecture for diverse 3D perception tasks. Our Height Association Transformer module leverages radar features already in the perspective view to produce more robust and accurate depth predictions. HyDRa achieves a new state-of-the-art for camera-radar fusion of 64.2 NDS (+1.8) and 58.4 AMOTA (+1.5) on the public nuScenes dataset.
  arXiv  Detail & Related papers  (2024-03-12T15:28:51Z)
• Instance-aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning [93.71280187657831]
  Camera-based bird-eye-view (BEV) perception paradigm has made significant progress in the autonomous driving field. We propose IA-BEV, which integrates image-plane instance awareness into the depth estimation process within a BEV-based detector.
  arXiv  Detail & Related papers  (2023-12-13T09:24:42Z)
• BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection [47.7933708173225]
  Recently, the rise of query-based Transformer decoders is reshaping camera-based 3D object detection. This paper introduces a "modernized" dense BEV framework dubbed BEVNeXt. On the nuScenes benchmark, BEVNeXt outperforms both BEV-based and query-based frameworks.
  arXiv  Detail & Related papers  (2023-12-04T07:35:02Z)
• Multi-camera Bird's Eye View Perception for Autonomous Driving [17.834495597639805]
  It is essential to produce perception outputs in 3D to enable the spatial reasoning of other agents and structures. The most basic approach to achieving the desired BEV representation from a camera image is IPM, assuming a flat ground surface. More recent approaches use deep neural networks to output directly in BEV space.
  arXiv  Detail & Related papers  (2023-09-16T19:12:05Z)
• OA-BEV: Bringing Object Awareness to Bird's-Eye-View Representation for Multi-Camera 3D Object Detection [78.38062015443195]
  OA-BEV is a network that can be plugged into the BEV-based 3D object detection framework. Our method achieves consistent improvements over the BEV-based baselines in terms of both average precision and nuScenes detection score.
  arXiv  Detail & Related papers  (2023-01-13T06:02:31Z)
• TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning [7.6887888234987125]
  We propose a learning scheme of Target Inner-Geometry from the LiDAR modality into camera-based BEV detectors. TiG-BEV can effectively boost BEVDepth by +2.3% NDS and +2.4% mAP, along with BEVDet by +9.1% NDS and +10.3% mAP on nuScenes val set.
  arXiv  Detail & Related papers  (2022-12-28T17:53:43Z)
• BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation [105.96557764248846]
  We introduce BEVFusion, a generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view representation space. It achieves 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9x lower cost.
  arXiv  Detail & Related papers  (2022-05-26T17:59:35Z)
• ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection [69.68263074432224]
  We present a novel framework named ZoomNet for stereo imagery-based 3D detection. The pipeline of ZoomNet begins with an ordinary 2D object detection model which is used to obtain pairs of left-right bounding boxes. To further exploit the abundant texture cues in RGB images for more accurate disparity estimation, we introduce a conceptually straight-forward module -- adaptive zooming.
  arXiv  Detail & Related papers  (2020-03-01T17:18:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.