DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

• URL: http://arxiv.org/abs/2110.06922v1
• Date: Wed, 13 Oct 2021 17:59:35 GMT
• Title: DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
• Authors: Yue Wang and Vitor Guizilini and Tianyuan Zhang and Yilun Wang and Hang Zhao and Justin Solomon
• Abstract summary: We introduce a framework for multi-camera 3D object detection. Our method manipulates predictions directly in 3D space. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.
• Score: 43.02373021724797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.

Related papers

• CVCP-Fusion: On Implicit Depth Estimation for 3D Bounding Box Prediction [2.0375637582248136]
  Cross-View Center Point-Fusion is a state-of-the-art model to perform 3D object detection. Our architecture utilizes aspects from previously established algorithms, Cross-View Transformers and CenterPoint.
  arXiv  Detail & Related papers  (2024-10-15T02:55:07Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• Monocular 3D Object Detection with Depth from Motion [74.29588921594853]
  We take advantage of camera ego-motion for accurate object depth estimation and detection. Our framework, named Depth from Motion (DfM), then uses the established geometry to lift 2D image features to the 3D space and detects 3D objects thereon. Our framework outperforms state-of-the-art methods by a large margin on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-07-26T15:48:46Z)
• AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection [15.244852122106634]
  We propose an approach for incorporating the shape-aware 2D/3D constraints into the 3D detection framework. Specifically, we employ the deep neural network to learn distinguished 2D keypoints in the 2D image domain. For generating the ground truth of 2D/3D keypoints, an automatic model-fitting approach has been proposed.
  arXiv  Detail & Related papers  (2021-08-25T08:50:06Z)
• PLUME: Efficient 3D Object Detection from Stereo Images [95.31278688164646]
  Existing methods tackle the problem in two steps: first depth estimation is performed, a pseudo LiDAR point cloud representation is computed from the depth estimates, and then object detection is performed in 3D space. We propose a model that unifies these two tasks in the same metric space. Our approach achieves state-of-the-art performance on the challenging KITTI benchmark, with significantly reduced inference time compared with existing methods.
  arXiv  Detail & Related papers  (2021-01-17T05:11:38Z)
• Single-Shot 3D Detection of Vehicles from Monocular RGB Images via Geometry Constrained Keypoints in Real-Time [6.82446891805815]
  We propose a novel 3D single-shot object detection method for detecting vehicles in monocular RGB images. Our approach lifts 2D detections to 3D space by predicting additional regression and classification parameters. We test our approach on different datasets for autonomous driving and evaluate it using the challenging KITTI 3D Object Detection and the novel nuScenes Object Detection benchmarks.
  arXiv  Detail & Related papers  (2020-06-23T15:10:19Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)
• BirdNet+: End-to-End 3D Object Detection in LiDAR Bird's Eye View [117.44028458220427]
  On-board 3D object detection in autonomous vehicles often relies on geometry information captured by LiDAR devices. We present a fully end-to-end 3D object detection framework that can infer oriented 3D boxes solely from BEV images.
  arXiv  Detail & Related papers  (2020-03-09T15:08:40Z)
• SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation [3.1542695050861544]
  Estimating 3D orientation and translation of objects is essential for infrastructure-less autonomous navigation and driving. We propose a novel 3D object detection method, named SMOKE, that combines a single keypoint estimate with regressed 3D variables. Despite of its structural simplicity, our proposed SMOKE network outperforms all existing monocular 3D detection methods on the KITTI dataset.
  arXiv  Detail & Related papers  (2020-02-24T08:15:36Z)
• RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving [26.216609821525676]
  Most successful 3D detectors take the projection constraint from the 3D bounding box to the 2D box as an important component. Our method predicts the nine perspective keypoints of a 3D bounding box in image space, and then utilize the geometric relationship of 3D and 2D perspectives to recover the dimension, location, and orientation in 3D space. Our method is the first real-time system for monocular image 3D detection while achieves state-of-the-art performance on the KITTI benchmark.
  arXiv  Detail & Related papers  (2020-01-10T08:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.