Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion

• URL: http://arxiv.org/abs/2211.10581v1
• Date: Sat, 19 Nov 2022 04:20:57 GMT
• Title: Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion
• Authors: Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, Zhizhong Su
• Abstract summary: We introduce Sparse4D, which does the iterative refinement of anchor boxes via sparsely sampling and fusing spatial-temporal features. In experiment, our method outperforms all sparse based methods and most BEV based methods on detection task in the nuScenes dataset.
• Score: 14.15155927539293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bird-eye-view (BEV) based methods have made great progress recently in multi-view 3D detection task. Comparing with BEV based methods, sparse based methods lag behind in performance, but still have lots of non-negligible merits. To push sparse 3D detection further, in this work, we introduce a novel method, named Sparse4D, which does the iterative refinement of anchor boxes via sparsely sampling and fusing spatial-temporal features. (1) Sparse 4D Sampling: for each 3D anchor, we assign multiple 4D keypoints, which are then projected to multi-view/scale/timestamp image features to sample corresponding features; (2) Hierarchy Feature Fusion: we hierarchically fuse sampled features of different view/scale, different timestamp and different keypoints to generate high-quality instance feature. In this way, Sparse4D can efficiently and effectively achieve 3D detection without relying on dense view transformation nor global attention, and is more friendly to edge devices deployment. Furthermore, we introduce an instance-level depth reweight module to alleviate the ill-posed issue in 3D-to-2D projection. In experiment, our method outperforms all sparse based methods and most BEV based methods on detection task in the nuScenes dataset.

Related papers

• Fully Sparse Fusion for 3D Object Detection [69.32694845027927]
  Currently prevalent multimodal 3D detection methods are built upon LiDAR-based detectors that usually use dense Bird's-Eye-View feature maps. Fully sparse architecture is gaining attention as they are highly efficient in long-range perception. In this paper, we study how to effectively leverage image modality in the emerging fully sparse architecture.
  arXiv  Detail & Related papers  (2023-04-24T17:57:43Z)
• Scatter Points in Space: 3D Detection from Multi-view Monocular Images [8.71944437852952]
  3D object detection from monocular image(s) is a challenging and long-standing problem of computer vision. Recent methods tend to aggregate multiview feature by sampling regular 3D grid densely in space. We propose a learnable keypoints sampling method, which scatters pseudo surface points in 3D space, in order to keep data sparsity.
  arXiv  Detail & Related papers  (2022-08-31T09:38:05Z)
• VPIT: Real-time Embedded Single Object 3D Tracking Using Voxel Pseudo Images [90.60881721134656]
  We propose a novel voxel-based 3D single object tracking (3D SOT) method called Voxel Pseudo Image Tracking (VPIT) Experiments on KITTI Tracking dataset show that VPIT is the fastest 3D SOT method and maintains competitive Success and Precision values.
  arXiv  Detail & Related papers  (2022-06-06T14:02:06Z)
• FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection [3.330229314824913]
  We present FCAF3D - a first-in-class fully convolutional anchor-free indoor 3D object detection method. It is a simple yet effective method that uses a voxel representation of a point cloud and processes voxels with sparse convolutions. It can handle large-scale scenes with minimal runtime through a single fully convolutional feed-forward pass.
  arXiv  Detail & Related papers  (2021-12-01T07:28:52Z)
• Soft Expectation and Deep Maximization for Image Feature Detection [68.8204255655161]
  We propose SEDM, an iterative semi-supervised learning process that flips the question and first looks for repeatable 3D points, then trains a detector to localize them in image space. Our results show that this new model trained using SEDM is able to better localize the underlying 3D points in a scene.
  arXiv  Detail & Related papers  (2021-04-21T00:35:32Z)
• PLUME: Efficient 3D Object Detection from Stereo Images [95.31278688164646]
  Existing methods tackle the problem in two steps: first depth estimation is performed, a pseudo LiDAR point cloud representation is computed from the depth estimates, and then object detection is performed in 3D space. We propose a model that unifies these two tasks in the same metric space. Our approach achieves state-of-the-art performance on the challenging KITTI benchmark, with significantly reduced inference time compared with existing methods.
  arXiv  Detail & Related papers  (2021-01-17T05:11:38Z)
• Generative Sparse Detection Networks for 3D Single-shot Object Detection [43.91336826079574]
  3D object detection has been widely studied due to its potential applicability to many promising areas such as robotics and augmented reality. Yet, the sparse nature of the 3D data poses unique challenges to this task. We propose Generative Sparse Detection Network (GSDN), a fully-convolutional single-shot sparse detection network.
  arXiv  Detail & Related papers  (2020-06-22T15:54:24Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)
• DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes [54.239416488865565]
  We propose a fast single-stage 3D object detection method for LIDAR data. The core novelty of our method is a fast, single-pass architecture that both detects objects in 3D and estimates their shapes. We find that our proposed method achieves state-of-the-art results by 5% on object detection in ScanNet scenes, and it gets top results by 3.4% in the Open dataset.
  arXiv  Detail & Related papers  (2020-04-02T17:48:50Z)
• RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving [26.216609821525676]
  Most successful 3D detectors take the projection constraint from the 3D bounding box to the 2D box as an important component. Our method predicts the nine perspective keypoints of a 3D bounding box in image space, and then utilize the geometric relationship of 3D and 2D perspectives to recover the dimension, location, and orientation in 3D space. Our method is the first real-time system for monocular image 3D detection while achieves state-of-the-art performance on the KITTI benchmark.
  arXiv  Detail & Related papers  (2020-01-10T08:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.