SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection

• URL: http://arxiv.org/abs/2403.07284v2
• Date: Wed, 10 Jul 2024 05:20:23 GMT
• Title: SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection
• Abstract summary: Sparse 3D detectors have received significant attention since the query-based paradigm embraces low latency without explicit dense BEV feature construction. Here, we present a high-performance fully sparse detector for end-to-end multi-modality 3D object detection. By the time of paper submission, SparseLIF achieves state-of-the-art performance on the nuScenes dataset, ranking 1st on both validation set and test benchmark.
• Score: 5.916444315216411
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sparse 3D detectors have received significant attention since the query-based paradigm embraces low latency without explicit dense BEV feature construction. However, these detectors achieve worse performance than their dense counterparts. In this paper, we find the key to bridging the performance gap is to enhance the awareness of rich representations in two modalities. Here, we present a high-performance fully sparse detector for end-to-end multi-modality 3D object detection. The detector, termed SparseLIF, contains three key designs, which are (1) Perspective-Aware Query Generation (PAQG) to generate high-quality 3D queries with perspective priors, (2) RoI-Aware Sampling (RIAS) to further refine prior queries by sampling RoI features from each modality, (3) Uncertainty-Aware Fusion (UAF) to precisely quantify the uncertainty of each sensor modality and adaptively conduct final multi-modality fusion, thus achieving great robustness against sensor noises. By the time of paper submission, SparseLIF achieves state-of-the-art performance on the nuScenes dataset, ranking 1st on both validation set and test benchmark, outperforming all state-of-the-art 3D object detectors by a notable margin.

Related papers

• ALIGN: Advanced Query Initialization with LiDAR-Image Guidance for Occlusion-Robust 3D Object Detection [16.336860116706088]
  We propose ALIGN, a novel approach for object-aware query initialization.<n>Our model consists of three key components: (i) Occlusion-aware Center Estimation (OCE), which integrates LiDAR geometry and image semantics.<n>Our experiments on the nuScenes benchmark demonstrate that ALIGN consistently improves performance across multiple state-of-the-art detectors.
  arXiv  Detail & Related papers  (2025-12-20T02:51:00Z)
• StereoMV2D: A Sparse Temporal Stereo-Enhanced Framework for Robust Multi-View 3D Object Detection [31.8104389684728]
  We propose StereoMV2D, a unified framework that integrates temporal stereo modeling into the 2D detection-guided multi-view 3D detector.<n>By exploiting cross-temporal disparities of the same object across adjacent frames, StereoMV2D enhances depth perception and refines the query priors.<n>Experiments on the nuScenes and Argoverse 2 datasets demonstrate that StereoMV2D achieves superior detection performance without incurring significant computational overhead.
  arXiv  Detail & Related papers  (2025-12-19T14:25:46Z)
• Real-IAD D3: A Real-World 2D/Pseudo-3D/3D Dataset for Industrial Anomaly Detection [53.2590751089607]
  Real-IAD D3 is a high-precision multimodal dataset that incorporates an additional pseudo3D modality generated through photometric stereo. We introduce an effective approach that integrates RGB, point cloud, and pseudo-3D depth information to leverage the complementary strengths of each modality. Our experiments highlight the importance of these modalities in boosting detection robustness and overall IAD performance.
  arXiv  Detail & Related papers  (2025-04-19T08:05:47Z)
• MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detection [28.319440934322728]
  MV2DFusion is a multi-modal detection framework that integrates the strengths of both worlds through an advanced query-based fusion mechanism. Our framework's flexibility allows it to integrate with any image and point cloud-based detectors, showcasing its adaptability and potential for future advancements.
  arXiv  Detail & Related papers  (2024-08-12T06:46:05Z)
• SEED: A Simple and Effective 3D DETR in Point Clouds [72.74016394325675]
  We argue that the main challenges are challenging due to the high sparsity and uneven distribution of point clouds. We propose a simple and effective 3D DETR method (SEED) for detecting 3D objects from point clouds.
  arXiv  Detail & Related papers  (2024-07-15T14:21:07Z)
• Multi-View Attentive Contextualization for Multi-View 3D Object Detection [19.874148893464607]
  We present Multi-View Attentive Contextualization (MvACon), a simple yet effective method for improving 2D-to-3D feature lifting in query-based 3D (MV3D) object detection. In experiments, the proposed MvACon is thoroughly tested on the nuScenes benchmark, using both the BEVFormer and its recent 3D deformable attention (DFA3D) variant, as well as the PETR.
  arXiv  Detail & Related papers  (2024-05-20T17:37:10Z)
• Fully Sparse Fusion for 3D Object Detection [69.32694845027927]
  Currently prevalent multimodal 3D detection methods are built upon LiDAR-based detectors that usually use dense Bird's-Eye-View feature maps. Fully sparse architecture is gaining attention as they are highly efficient in long-range perception. In this paper, we study how to effectively leverage image modality in the emerging fully sparse architecture.
  arXiv  Detail & Related papers  (2023-04-24T17:57:43Z)
• Multi-level and multi-modal feature fusion for accurate 3D object detection in Connected and Automated Vehicles [0.8701566919381223]
  This paper presents a Deep Neural Network based 3D object detection model that leverages a three-stage feature extractor. The proposed feature extractor extracts high-level features from two input sensory modalities and recovers the important features discarded during the convolutional process. The novel fusion scheme effectively fuses features across sensory modalities and convolutional layers to find the best representative global features.
  arXiv  Detail & Related papers  (2022-12-15T00:25:05Z)
• D-Align: Dual Query Co-attention Network for 3D Object Detection Based on Multi-frame Point Cloud Sequence [8.21339007493213]
  Conventional 3D object detectors detect objects using a set of points acquired over a fixed duration. Recent studies have shown that the performance of object detection can be further enhanced by utilizing point cloud sequences. We propose D-Align, which can effectively produce strong bird's-eye-view (BEV) features by aligning and aggregating the features obtained from a sequence of point sets.
  arXiv  Detail & Related papers  (2022-09-30T20:41:25Z)
• EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection [56.03081616213012]
  We propose EPNet++ for multi-modal 3D object detection by introducing a novel Cascade Bi-directional Fusion(CB-Fusion) module. The proposed CB-Fusion module boosts the plentiful semantic information of point features with the image features in a cascade bi-directional interaction fusion manner. The experiment results on the KITTI, JRDB and SUN-RGBD datasets demonstrate the superiority of EPNet++ over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-21T10:48:34Z)
• Improving 3D Object Detection with Channel-wise Transformer [58.668922561622466]
  We propose a two-stage 3D object detection framework (CT3D) with minimal hand-crafted design. CT3D simultaneously performs proposal-aware embedding and channel-wise context aggregation. It achieves the AP of 81.77% in the moderate car category on the KITTI test 3D detection benchmark.
  arXiv  Detail & Related papers  (2021-08-23T02:03:40Z)
• M3DSSD: Monocular 3D Single Stage Object Detector [82.25793227026443]
  We propose a Monocular 3D Single Stage object Detector (M3DSSD) with feature alignment and asymmetric non-local attention. The proposed M3DSSD achieves significantly better performance than the monocular 3D object detection methods on the KITTI dataset.
  arXiv  Detail & Related papers  (2021-03-24T13:09:11Z)
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection [103.5060007382646]
  We propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. We design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution.
  arXiv  Detail & Related papers  (2020-12-20T18:43:41Z)
• Boundary-Aware Dense Feature Indicator for Single-Stage 3D Object Detection from Point Clouds [32.916690488130506]
  We propose a universal module that helps 3D detectors focus on the densest region of the point clouds in a boundary-aware manner. Experiments on KITTI dataset show that DENFI improves the performance of the baseline single-stage detector remarkably.
  arXiv  Detail & Related papers  (2020-04-01T01:21:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.