Zoo3D: Zero-Shot 3D Object Detection at Scene Level

• URL: http://arxiv.org/abs/2511.20253v1
• Date: Tue, 25 Nov 2025 12:29:06 GMT
• Title: Zoo3D: Zero-Shot 3D Object Detection at Scene Level
• Authors: Andrey Lemeshko, Bulat Gabdullin, Nikita Drozdov, Anton Konushin, Danila Rukhovich, Maksim Kolodiazhnyi,
• Abstract summary: Zoo3D is the first training-free 3D object detection framework.<n>Our method constructs 3D bounding boxes via graph clustering of 2D instance masks.<n>We extend Zoo3D beyond point clouds to work directly with posed and even unposed images.
• Score: 7.756226313216256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D object detection is fundamental for spatial understanding. Real-world environments demand models capable of recognizing diverse, previously unseen objects, which remains a major limitation of closed-set methods. Existing open-vocabulary 3D detectors relax annotation requirements but still depend on training scenes, either as point clouds or images. We take this a step further by introducing Zoo3D, the first training-free 3D object detection framework. Our method constructs 3D bounding boxes via graph clustering of 2D instance masks, then assigns semantic labels using a novel open-vocabulary module with best-view selection and view-consensus mask generation. Zoo3D operates in two modes: the zero-shot Zoo3D$_0$, which requires no training at all, and the self-supervised Zoo3D$_1$, which refines 3D box prediction by training a class-agnostic detector on Zoo3D$_0$-generated pseudo labels. Furthermore, we extend Zoo3D beyond point clouds to work directly with posed and even unposed images. Across ScanNet200 and ARKitScenes benchmarks, both Zoo3D$_0$ and Zoo3D$_1$ achieve state-of-the-art results in open-vocabulary 3D object detection. Remarkably, our zero-shot Zoo3D$_0$ outperforms all existing self-supervised methods, hence demonstrating the power and adaptability of training-free, off-the-shelf approaches for real-world 3D understanding. Code is available at https://github.com/col14m/zoo3d .

Related papers

• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight [105.9472902251177]
  We present a VLM-native recipe that casts 3D detection as a next-token prediction problem.<n>Our model achieves state-of-the-art results, with 49.89 AP_3D, surpassing the previous best by +15.51 absolute improvement.
  arXiv  Detail & Related papers  (2025-11-25T18:59:45Z)
• 3D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detection [62.57179069154312]
  We introduce the first end-to-end 3D Monocular Open-set Object Detector (3D-MOOD)<n>We lift the open-set 2D detection into 3D space through our designed 3D bounding box head.<n>We condition the object queries with geometry prior and overcome the generalization for 3D estimation across diverse scenes.
  arXiv  Detail & Related papers  (2025-07-31T13:56:41Z)
• Just Add Geometry: Gradient-Free Open-Vocabulary 3D Detection Without Human-in-the-Loop [0.0]
  2D vision-language models trained on web-scale image-text pairs exhibit rich semantic understanding and support open-vocabulary detection.<n>We leverage the maturity and category diversity of 2D foundation models to perform 3D object detection without any human-annotated 3D labels.<n>Our results highlight the untapped potential of 2D foundation models for scalable 3D perception.
  arXiv  Detail & Related papers  (2025-07-06T15:00:13Z)
• Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data [57.53523870705433]
  We propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det. OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. It employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors.
  arXiv  Detail & Related papers  (2024-11-23T21:37:21Z)
• OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding [54.981605111365056]
  This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding.<n>Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing.
  arXiv  Detail & Related papers  (2024-06-04T07:42:33Z)
• Uni3D: Exploring Unified 3D Representation at Scale [66.26710717073372]
  We present Uni3D, a 3D foundation model to explore the unified 3D representation at scale. Uni3D uses a 2D ViT end-to-end pretrained to align the 3D point cloud features with the image-text aligned features. We show that the strong Uni3D representation also enables applications such as 3D painting and retrieval in the wild.
  arXiv  Detail & Related papers  (2023-10-10T16:49:21Z)
• Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection [24.871590175483096]
  Point cloud-based open-vocabulary 3D object detection aims to detect 3D categories that do not have ground-truth annotations in the training set. Previous approaches leverage large-scale richly-annotated image datasets as a bridge between 3D and category semantics. We propose Object2Scene, the first approach that leverages large-scale large-vocabulary 3D object datasets to augment existing 3D scene datasets for open-vocabulary 3D object detection.
  arXiv  Detail & Related papers  (2023-09-18T03:31:53Z)
• OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation [32.508069732371105]
  OpenIns3D is a new 3D-input-only framework for 3D open-vocabulary scene understanding. It achieves state-of-the-art performance across a wide range of 3D open-vocabulary tasks.
  arXiv  Detail & Related papers  (2023-09-01T17:59:56Z)
• Gait Recognition in the Wild with Dense 3D Representations and A Benchmark [86.68648536257588]
  Existing studies for gait recognition are dominated by 2D representations like the silhouette or skeleton of the human body in constrained scenes. This paper aims to explore dense 3D representations for gait recognition in the wild. We build the first large-scale 3D representation-based gait recognition dataset, named Gait3D.
  arXiv  Detail & Related papers  (2022-04-06T03:54:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.