A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding

• URL: http://arxiv.org/abs/2508.01197v1
• Date: Sat, 02 Aug 2025 05:05:50 GMT
• Title: A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding
• Authors: Zhan Shi, Song Wang, Junbo Chen, Jianke Zhu,
• Abstract summary: We introduce a benchmark for 3D occupancy grounding in challenging outdoor scenes.<n>Built on the nuScenes dataset, it integrates natural language with voxel-level occupancy annotations.<n>We also propose GroundingOcc, an end-to-end model designed for 3D occupancy grounding through multi-modal learning.
• Score: 21.59984961930343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual grounding aims to identify objects or regions in a scene based on natural language descriptions, essential for spatially aware perception in autonomous driving. However, existing visual grounding tasks typically depend on bounding boxes that often fail to capture fine-grained details. Not all voxels within a bounding box are occupied, resulting in inaccurate object representations. To address this, we introduce a benchmark for 3D occupancy grounding in challenging outdoor scenes. Built on the nuScenes dataset, it integrates natural language with voxel-level occupancy annotations, offering more precise object perception compared to the traditional grounding task. Moreover, we propose GroundingOcc, an end-to-end model designed for 3D occupancy grounding through multi-modal learning. It combines visual, textual, and point cloud features to predict object location and occupancy information from coarse to fine. Specifically, GroundingOcc comprises a multimodal encoder for feature extraction, an occupancy head for voxel-wise predictions, and a grounding head to refine localization. Additionally, a 2D grounding module and a depth estimation module enhance geometric understanding, thereby boosting model performance. Extensive experiments on the benchmark demonstrate that our method outperforms existing baselines on 3D occupancy grounding. The dataset is available at https://github.com/RONINGOD/GroundingOcc.

Related papers

• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation [54.04601077224252]
  Embodied scene understanding requires not only comprehending visual-spatial information but also determining where to explore next in the 3D physical world.<n>underlinetextbf3D vision-language learning enables embodied agents to effectively explore and understand their environment.<n>model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images.
  arXiv  Detail & Related papers  (2025-07-05T14:15:52Z)
• From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes [16.38713257618971]
  Anywhere3D-Bench is a holistic 3D visual grounding benchmark consisting of 2,632 referring expression-3D bounding box pairs.<n>We assess a range of state-of-the-art 3D visual grounding methods alongside large language models.
  arXiv  Detail & Related papers  (2025-06-05T11:28:02Z)
• SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models [9.568997654206823]
  SORT3D is an approach that utilizes rich object attributes from 2D data and merges as-based spatial reasoning toolbox with the ability of large language models.<n>We show that SORT3D achieves state-of-the-art performance on complex view-dependent grounding tasks on two benchmarks.<n>We also implement the pipeline to run real-time on an autonomous vehicle and demonstrate that our approach can be used for object-goal navigation on previously unseen real-world environments.
  arXiv  Detail & Related papers  (2025-04-25T20:24:11Z)
• Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection [54.78470057491049]
  Occupancy has emerged as a promising alternative for 3D scene perception.<n>We introduce object-centric occupancy as a supplement to object bboxes.<n>We show that our occupancy features significantly enhance the detection results of state-of-the-art 3D object detectors.
  arXiv  Detail & Related papers  (2024-12-06T16:12:38Z)
• Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding [56.00186960144545]
  3D visual grounding is the task of localizing the object in a 3D scene which is referred by a description in natural language. We propose a dense 3D grounding network, featuring four novel stand-alone modules that aim to improve grounding performance.
  arXiv  Detail & Related papers  (2023-09-08T19:27:01Z)
• SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving [98.74706005223685]
  3D scene understanding plays a vital role in vision-based autonomous driving. We propose a SurroundOcc method to predict the 3D occupancy with multi-camera images.
  arXiv  Detail & Related papers  (2023-03-16T17:59:08Z)
• Det6D: A Ground-Aware Full-Pose 3D Object Detector for Improving Terrain Robustness [1.4620086904601473]
  We propose Det6D, the first full-degree-of-freedom 3D object detector without spatial and postural limitations. To predict full-degree poses, including pitch and roll, we design a ground-aware orientation branch. Experiments on various datasets demonstrate the effectiveness and robustness of our method in different terrains.
  arXiv  Detail & Related papers  (2022-07-19T17:12:48Z)
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding [72.7618059299306]
  We develop a spatial-language model for a 3D visual grounding problem. We show that our model performs competitively on visio-linguistic datasets proposed by ReferIt3D.
  arXiv  Detail & Related papers  (2021-07-07T18:55:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.