Open Vocabulary Monocular 3D Object Detection

• URL: http://arxiv.org/abs/2411.16833v1
• Date: Mon, 25 Nov 2024 18:59:17 GMT
• Title: Open Vocabulary Monocular 3D Object Detection
• Authors: Jin Yao, Hao Gu, Xuweiyi Chen, Jiayun Wang, Zezhou Cheng,
• Abstract summary: We pioneer the study of open-vocabulary monocular 3D object detection, a novel task that aims to detect and localize objects in 3D space from a single RGB image. We introduce a class-agnostic approach that leverages open-vocabulary 2D detectors and lifts 2D bounding boxes into 3D space. Our approach decouples the recognition and localization of objects in 2D from the task of estimating 3D bounding boxes, enabling generalization across unseen categories.
• Score: 10.424711580213616
• License:
• Abstract: In this work, we pioneer the study of open-vocabulary monocular 3D object detection, a novel task that aims to detect and localize objects in 3D space from a single RGB image without limiting detection to a predefined set of categories. We formalize this problem, establish baseline methods, and introduce a class-agnostic approach that leverages open-vocabulary 2D detectors and lifts 2D bounding boxes into 3D space. Our approach decouples the recognition and localization of objects in 2D from the task of estimating 3D bounding boxes, enabling generalization across unseen categories. Additionally, we propose a target-aware evaluation protocol to address inconsistencies in existing datasets, improving the reliability of model performance assessment. Extensive experiments on the Omni3D dataset demonstrate the effectiveness of the proposed method in zero-shot 3D detection for novel object categories, validating its robust generalization capabilities. Our method and evaluation protocols contribute towards the development of open-vocabulary object detection models that can effectively operate in real-world, category-diverse environments.

Related papers

• Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data [57.53523870705433]
  We propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det. OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. It employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors.
  arXiv  Detail & Related papers  (2024-11-23T21:37:21Z)
• Open-Set 3D object detection in LiDAR data as an Out-of-Distribution problem [6.131026007721572]
  3D Object Detection from LiDAR data has achieved industry-ready performance in controlled environments. Our work redefines the open-set 3D Object Detection problem in LiDAR data as an Out-Of-Distribution (OOD) problem to detect outlier objects.
  arXiv  Detail & Related papers  (2024-10-31T09:29:55Z)
• Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments [67.83787474506073]
  We tackle the limitations of current LiDAR-based 3D object detection systems. We introduce a universal textscFind n' Propagate approach for 3D OV tasks. We achieve up to a 3.97-fold increase in Average Precision (AP) for novel object classes.
  arXiv  Detail & Related papers  (2024-03-20T12:51:30Z)
• Scalable Vision-Based 3D Object Detection and Monocular Depth Estimation for Autonomous Driving [5.347428263669927]
  This dissertation is a multifaceted contribution to the advancement of vision-based 3D perception technologies. In the first segment, the thesis introduces structural enhancements to both monocular and stereo 3D object detection algorithms. The second segment is devoted to data-driven strategies and their real-world applications in 3D vision detection.
  arXiv  Detail & Related papers  (2024-03-04T13:42:54Z)
• OriCon3D: Effective 3D Object Detection using Orientation and Confidence [0.0]
  We propose an advanced methodology for the detection of 3D objects from a single image. We use a deep convolutional neural network-based 3D object weighted orientation regression paradigm. Our approach significantly improves the accuracy of 3D object pose determination, surpassing baseline methodologies.
  arXiv  Detail & Related papers  (2023-04-27T19:52:47Z)
• CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection [57.44434974289945]
  We propose Contextualized Multi-Stage Refinement for 3D Object Detection (CMR3D) framework. Our framework takes a 3D scene as input and strives to explicitly integrate useful contextual information of the scene. In addition to 3D object detection, we investigate the effectiveness of our framework for the problem of 3D object counting.
  arXiv  Detail & Related papers  (2022-09-13T05:26:09Z)
• Shape-Aware Monocular 3D Object Detection [15.693199934120077]
  A single-stage monocular 3D object detection model is proposed. The detection largely avoids interference from irrelevant regions surrounding the target objects. A novel evaluation metric, namely average depth similarity (ADS) is proposed for the monocular 3D object detection models.
  arXiv  Detail & Related papers  (2022-04-19T07:43:56Z)
• Homography Loss for Monocular 3D Object Detection [54.04870007473932]
  A differentiable loss function, termed as Homography Loss, is proposed to achieve the goal, which exploits both 2D and 3D information. Our method yields the best performance compared with the other state-of-the-arts by a large margin on KITTI 3D datasets.
  arXiv  Detail & Related papers  (2022-04-02T03:48:03Z)
• Delving into Localization Errors for Monocular 3D Object Detection [85.77319416168362]
  Estimating 3D bounding boxes from monocular images is an essential component in autonomous driving. In this work, we quantify the impact introduced by each sub-task and find the localization error' is the vital factor in restricting monocular 3D detection.
  arXiv  Detail & Related papers  (2021-03-30T10:38:01Z)
• Reinforced Axial Refinement Network for Monocular 3D Object Detection [160.34246529816085]
  Monocular 3D object detection aims to extract the 3D position and properties of objects from a 2D input image. Conventional approaches sample 3D bounding boxes from the space and infer the relationship between the target object and each of them, however, the probability of effective samples is relatively small in the 3D space. We propose to start with an initial prediction and refine it gradually towards the ground truth, with only one 3d parameter changed in each step. This requires designing a policy which gets a reward after several steps, and thus we adopt reinforcement learning to optimize it.
  arXiv  Detail & Related papers  (2020-08-31T17:10:48Z)
• SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation [3.1542695050861544]
  Estimating 3D orientation and translation of objects is essential for infrastructure-less autonomous navigation and driving. We propose a novel 3D object detection method, named SMOKE, that combines a single keypoint estimate with regressed 3D variables. Despite of its structural simplicity, our proposed SMOKE network outperforms all existing monocular 3D detection methods on the KITTI dataset.
  arXiv  Detail & Related papers  (2020-02-24T08:15:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.