MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer for Autonomous Driving

• URL: http://arxiv.org/abs/2210.15316v1
• Date: Thu, 27 Oct 2022 10:55:15 GMT
• Title: MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer for Autonomous Driving
• Authors: Gopi Krishna Erabati and Helder Araujo
• Abstract summary: We propose MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer architecture to fuse image and LiDAR features to improve the detection accuracy. Our end-to-end single-stage, anchor-free and NMS-free network takes in multi-view images and LiDAR point clouds and predicts 3D bounding boxes. MSF3DDETR network is trained end-to-end on the nuScenes dataset using Hungarian algorithm based bipartite matching and set-to-set loss inspired by DETR.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D object detection is a significant task for autonomous driving. Recently with the progress of vision transformers, the 2D object detection problem is being treated with the set-to-set loss. Inspired by these approaches on 2D object detection and an approach for multi-view 3D object detection DETR3D, we propose MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer architecture to fuse image and LiDAR features to improve the detection accuracy. Our end-to-end single-stage, anchor-free and NMS-free network takes in multi-view images and LiDAR point clouds and predicts 3D bounding boxes. Firstly, we link the object queries learnt from data to the image and LiDAR features using a novel MSF3DDETR cross-attention block. Secondly, the object queries interacts with each other in multi-head self-attention block. Finally, MSF3DDETR block is repeated for $L$ number of times to refine the object queries. The MSF3DDETR network is trained end-to-end on the nuScenes dataset using Hungarian algorithm based bipartite matching and set-to-set loss inspired by DETR. We present both quantitative and qualitative results which are competitive to the state-of-the-art approaches.

Related papers

• Sparse Points to Dense Clouds: Enhancing 3D Detection with Limited LiDAR Data [68.18735997052265]
  We propose a balanced approach that combines the advantages of monocular and point cloud-based 3D detection. Our method requires only a small number of 3D points, that can be obtained from a low-cost, low-resolution sensor. The accuracy of 3D detection improves by 20% compared to the state-of-the-art monocular detection methods.
  arXiv  Detail & Related papers  (2024-04-10T03:54:53Z)
• PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interest [65.48057241587398]
  PoIFusion is a framework to fuse information of RGB images and LiDAR point clouds at the points of interest (PoIs) Our approach maintains the view of each modality and obtains multi-modal features by computation-friendly projection and computation. We conducted extensive experiments on nuScenes and Argoverse2 datasets to evaluate our approach.
  arXiv  Detail & Related papers  (2024-03-14T09:28:12Z)
• Multi-Modal 3D Object Detection by Box Matching [109.43430123791684]
  We propose a novel Fusion network by Box Matching (FBMNet) for multi-modal 3D detection. With the learned assignments between 3D and 2D object proposals, the fusion for detection can be effectively performed by combing their ROI features.
  arXiv  Detail & Related papers  (2023-05-12T18:08:51Z)
• Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection [11.575945934519442]
  LiDAR and camera fusion techniques are promising for achieving 3D object detection in autonomous driving. Most multi-modal 3D object detection frameworks integrate semantic knowledge from 2D images into 3D LiDAR point clouds. We propose a general multi-modal fusion framework Multi-Sem Fusion (MSF) to fuse the semantic information from both the 2D image and 3D points scene parsing results.
  arXiv  Detail & Related papers  (2022-12-10T10:54:41Z)
• Bridged Transformer for Vision and Point Cloud 3D Object Detection [92.86856146086316]
  Bridged Transformer (BrT) is an end-to-end architecture for 3D object detection. BrT learns to identify 3D and 2D object bounding boxes from both points and image patches. We experimentally show that BrT surpasses state-of-the-art methods on SUN RGB-D and ScanNetV2 datasets.
  arXiv  Detail & Related papers  (2022-10-04T05:44:22Z)
• CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection [57.44434974289945]
  We propose Contextualized Multi-Stage Refinement for 3D Object Detection (CMR3D) framework. Our framework takes a 3D scene as input and strives to explicitly integrate useful contextual information of the scene. In addition to 3D object detection, we investigate the effectiveness of our framework for the problem of 3D object counting.
  arXiv  Detail & Related papers  (2022-09-13T05:26:09Z)
• FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection [15.641616738865276]
  We propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task. Especially, the FusionPainting framework consists of three main modules: a multi-modal semantic segmentation module, an adaptive attention-based semantic fusion module, and a 3D object detector. The effectiveness of the proposed framework has been verified on the large-scale nuScenes detection benchmark.
  arXiv  Detail & Related papers  (2021-06-23T14:53:22Z)
• Relation3DMOT: Exploiting Deep Affinity for 3D Multi-Object Tracking from View Aggregation [8.854112907350624]
  3D multi-object tracking plays a vital role in autonomous navigation. Many approaches detect objects in 2D RGB sequences for tracking, which is lack of reliability when localizing objects in 3D space. We propose a novel convolutional operation, named RelationConv, to better exploit the correlation between each pair of objects in the adjacent frames.
  arXiv  Detail & Related papers  (2020-11-25T16:14:40Z)
• CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection [13.986963122264633]
  We propose a novel Camera-LiDAR Object Candidates (CLOCs) fusion network. CLOCs fusion provides a low-complexity multi-modal fusion framework. We show that CLOCs ranks the highest among all the fusion-based methods in the official KITTI leaderboard.
  arXiv  Detail & Related papers  (2020-09-02T02:07:00Z)
• Cross-Modality 3D Object Detection [63.29935886648709]
  We present a novel two-stage multi-modal fusion network for 3D object detection. The whole architecture facilitates two-stage fusion. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.
  arXiv  Detail & Related papers  (2020-08-16T11:01:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.