TransCAR: Transformer-based Camera-And-Radar Fusion for 3D Object Detection

• URL: http://arxiv.org/abs/2305.00397v1
• Date: Sun, 30 Apr 2023 05:35:03 GMT
• Title: TransCAR: Transformer-based Camera-And-Radar Fusion for 3D Object Detection
• Authors: Su Pang, Daniel Morris, Hayder Radha
• Abstract summary: TransCAR is a Transformer-based Camera-And-Radar fusion solution for 3D object detection. Our model estimates a bounding box per query using set-to-set Hungarian loss.
• Score: 13.986963122264633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite radar's popularity in the automotive industry, for fusion-based 3D object detection, most existing works focus on LiDAR and camera fusion. In this paper, we propose TransCAR, a Transformer-based Camera-And-Radar fusion solution for 3D object detection. Our TransCAR consists of two modules. The first module learns 2D features from surround-view camera images and then uses a sparse set of 3D object queries to index into these 2D features. The vision-updated queries then interact with each other via transformer self-attention layer. The second module learns radar features from multiple radar scans and then applies transformer decoder to learn the interactions between radar features and vision-updated queries. The cross-attention layer within the transformer decoder can adaptively learn the soft-association between the radar features and vision-updated queries instead of hard-association based on sensor calibration only. Finally, our model estimates a bounding box per query using set-to-set Hungarian loss, which enables the method to avoid non-maximum suppression. TransCAR improves the velocity estimation using the radar scans without temporal information. The superior experimental results of our TransCAR on the challenging nuScenes datasets illustrate that our TransCAR outperforms state-of-the-art Camera-Radar fusion-based 3D object detection approaches.

Related papers

• RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network [34.45694077040797]
  We present a radar-camera fusion 3D object detection framework called BEEVDet. RadarBEVNet encodes sparse radar points into a dense bird's-eye-view feature. Our method achieves state-of-the-art radar-camera fusion results in 3D object detection, BEV semantic segmentation, and 3D multi-object tracking tasks.
  arXiv  Detail & Related papers  (2024-09-08T05:14:27Z)
• RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection [33.07575082922186]
  Three-dimensional object detection is one of the key tasks in autonomous driving. relying solely on cameras is difficult to achieve highly accurate and robust 3D object detection. radar-camera fusion 3D object detection method in the bird's eye view (BEV) RadarBEVNet consists of a dual-stream radar backbone and a Radar Cross-Section (RC) aware BEV encoder.
  arXiv  Detail & Related papers  (2024-03-25T06:02:05Z)
• RCM-Fusion: Radar-Camera Multi-Level Fusion for 3D Object Detection [15.686167262542297]
  We propose Radar-Camera Multi-level fusion (RCM-Fusion), which attempts to fuse both modalities at both feature and instance levels. For feature-level fusion, we propose a Radar Guided BEV which transforms camera features into precise BEV representations. For instance-level fusion, we propose a Radar Grid Point Refinement module that reduces localization error.
  arXiv  Detail & Related papers  (2023-07-17T07:22:25Z)
• Bi-LRFusion: Bi-Directional LiDAR-Radar Fusion for 3D Dynamic Object Detection [78.59426158981108]
  We introduce a bi-directional LiDAR-Radar fusion framework, termed Bi-LRFusion, to tackle the challenges and improve 3D detection for dynamic objects. We conduct extensive experiments on nuScenes and ORR datasets, and show that our Bi-LRFusion achieves state-of-the-art performance for detecting dynamic objects.
  arXiv  Detail & Related papers  (2023-06-02T10:57:41Z)
• Multi-Modal 3D Object Detection by Box Matching [109.43430123791684]
  We propose a novel Fusion network by Box Matching (FBMNet) for multi-modal 3D detection. With the learned assignments between 3D and 2D object proposals, the fusion for detection can be effectively performed by combing their ROI features.
  arXiv  Detail & Related papers  (2023-05-12T18:08:51Z)
• MVFusion: Multi-View 3D Object Detection with Semantic-aligned Radar and Camera Fusion [6.639648061168067]
  Multi-view radar-camera fused 3D object detection provides a farther detection range and more helpful features for autonomous driving. Current radar-camera fusion methods deliver kinds of designs to fuse radar information with camera data. We present MVFusion, a novel Multi-View radar-camera Fusion method to achieve semantic-aligned radar features.
  arXiv  Detail & Related papers  (2023-02-21T08:25:50Z)
• CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection [12.557361522985898]
  We propose a camera-radar matching network CramNet to fuse the sensor readings from camera and radar in a joint 3D space. Our method supports training with sensor modality dropout, which leads to robust 3D object detection, even when a camera or radar sensor suddenly malfunctions on a vehicle.
  arXiv  Detail & Related papers  (2022-10-17T17:18:47Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers [49.689566246504356]
  We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. TransFusion achieves state-of-the-art performance on large-scale datasets. We extend the proposed method to the 3D tracking task and achieve the 1st place in the leaderboard of nuScenes tracking.
  arXiv  Detail & Related papers  (2022-03-22T07:15:13Z)
• DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection [83.18142309597984]
  Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. We develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods.
  arXiv  Detail & Related papers  (2022-03-15T18:46:06Z)
• RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects [73.80316195652493]
  We tackle the problem of exploiting Radar for perception in the context of self-driving cars. We propose a new solution that exploits both LiDAR and Radar sensors for perception. Our approach, dubbed RadarNet, features a voxel-based early fusion and an attention-based late fusion.
  arXiv  Detail & Related papers  (2020-07-28T17:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.