Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

• URL: http://arxiv.org/abs/2602.22667v1
• Date: Thu, 26 Feb 2026 06:37:43 GMT
• Title: Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes
• Abstract summary: Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments.<n>We propose a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free)<n>Our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU.
• Score: 12.60608820766928
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.

Related papers

• DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation [53.011370226020695]
  We propose DINOde, an ODE-based framework that aligns CLIP text embeddings with the DINO visual manifold.<n>Our approach employs two complementary components: (i) Semantic Text Flow (STF), which evolves text embeddings toward the DINO manifold through a continuous ODE trajectory, and (ii) Global Context Flow (GCF), which refines the holistic image representation carried by DINO's CLS token.<n>By modeling alignment as a continuous trajectory, DINOde avoids the entanglement inherent in discrete projections and yields more robust cross-modal alignment.
  arXiv  Detail & Related papers  (2026-07-23T14:37:27Z)
• Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images [62.31892851760248]
  COVScene is a pose-free semantic Gaussian framework that couples renderable primitives with a dense semantic occupancy field.<n>We show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline.
  arXiv  Detail & Related papers  (2026-07-02T03:00:23Z)
• OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention [80.51557267896938]
  We present OpenGaFF, a novel framework for open-vocabulary 3D scene understanding built upon 3D Gaussian Splatting.<n>At the core of our method is a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance.<n>To further enforce object-level semantic consistency, we introduce a structured codebook that serves as a set of shared semantic primitives.
  arXiv  Detail & Related papers  (2026-05-07T12:10:07Z)
• FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction [11.503430067699442]
  FreeOcc is a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences.<n>FreeOcc operates without 3D annotations, pose ground truth, or any learning stage.<n>FreeOcc achieves over $2times$ improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods.
  arXiv  Detail & Related papers  (2026-04-30T17:05:56Z)
• LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding [9.377694035678948]
  We present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation.<n>This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS.<n>Our approach achieves state-of-the-art performance on Open Vocabulary 3D Object Retrieval and Point Cloud Understanding benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T20:48:06Z)
• HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulation [54.325346533275074]
  HeRO is a diffusion-based policy that couples geometry and semantics via hierarchical semantic fields.<n>In various tests, HeRO establishes a new state-of-the-art, improving success on Place Dual Shoes by 12.3% and averaging 6.5% gains across six challenging pose-aware tasks.
  arXiv  Detail & Related papers  (2026-02-21T12:29:10Z)
• ProFuse: Efficient Cross-View Context Fusion for Open-Vocabulary 3D Gaussian Splatting [1.1470070927586018]
  ProFuse is an efficient context-aware framework for open-vocabulary 3D scene understanding with 3D Gaussian Splatting (3DGS)<n>The pipeline enhances cross-view consistency and intra-mask cohesion within a direct registration setup.<n>ProFuse achieves strong open-vocabulary 3DGS understanding while completing semantic attachment in about five minutes per scene.
  arXiv  Detail & Related papers  (2026-01-08T09:20:46Z)
• SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation [114.57192386025373]
  SegSplat is a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding.<n>This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments.
  arXiv  Detail & Related papers  (2025-11-23T10:26:38Z)
• OpenInsGaussian: Open-vocabulary Instance Gaussian Segmentation with Context-aware Cross-view Fusion [89.98812408058336]
  We introduce textbfOpenInsGaussian, an textbfOpen-vocabulary textbfInstance textbfGaussian segmentation framework with Context-aware Cross-view Fusion.<n>OpenInsGaussian achieves state-of-the-art results in open-vocabulary 3D Gaussian segmentation, outperforming existing baselines by a large margin.
  arXiv  Detail & Related papers  (2025-10-21T03:24:12Z)
• OpenGS-Fusion: Open-Vocabulary Dense Mapping with Hybrid 3D Gaussian Splatting for Refined Object-Level Understanding [17.524454394142477]
  We present OpenGS-Fusion, an innovative open-vocabulary dense mapping framework that improves semantic modeling and refines object-level understanding.<n>We also introduce a novel multimodal language-guided approach named MLLM-Assisted Adaptive Thresholding, which refines the segmentation of 3D objects by adaptively adjusting similarity thresholds.<n>Our method outperforms existing methods in 3D object understanding and scene reconstruction quality, as well as showcasing its effectiveness in language-guided scene interaction.
  arXiv  Detail & Related papers  (2025-08-02T02:22:36Z)
• PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum [20.206273757144547]
  PGOV3D is a novel framework that introduces a Partial-to-Global curriculum for improving open-vocabulary 3D semantic segmentation.<n>We pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry.<n>In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex.
  arXiv  Detail & Related papers  (2025-06-30T08:13:07Z)
• Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding [58.38294408121273]
  We propose Cross-modal and Uncertainty-aware Agglomeration for Open-vocabulary 3D Scene Understanding dubbed CUA-O3D.<n>Our method addresses two key challenges: (1) incorporating semantic priors from VLMs alongside the geometric knowledge of spatially-aware vision foundation models, and (2) using a novel deterministic uncertainty estimation to capture model-specific uncertainties.
  arXiv  Detail & Related papers  (2025-03-20T20:58:48Z)
• GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane [53.388937705785025]
  3D open-vocabulary scene understanding is crucial for advancing augmented reality and robotic applications. We introduce GOI, a framework that integrates semantic features from 2D vision-language foundation models into 3D Gaussian Splatting (3DGS) Our method treats the feature selection process as a hyperplane division within the feature space, retaining only features that are highly relevant to the query.
  arXiv  Detail & Related papers  (2024-05-27T18:57:18Z)
• GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields [50.68719394443926]
  Generalizable Open-Vocabulary Neural Semantic Fields (GOV-NeSF) is a novel approach offering a generalizable implicit representation of 3D scenes with open-vocabulary semantics. GOV-NeSF exhibits state-of-the-art performance in both 2D and 3D open-vocabulary semantic segmentation.
  arXiv  Detail & Related papers  (2024-04-01T05:19:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.