Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images

• URL: http://arxiv.org/abs/2607.01633v1
• Date: Thu, 02 Jul 2026 03:00:23 GMT
• Title: Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images
• Abstract summary: COVScene is a pose-free semantic Gaussian framework that couples renderable primitives with a dense semantic occupancy field.<n>We show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline.
• Score: 62.31892851760248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Comprehensive 3D scene understanding from sparse, unposed images requires a model to recover renderable geometry, open-vocabulary semantics, and free/occupied 3D space without relying on external camera calibration. Recent feed-forward Gaussian methods improve pose-free reconstruction and semantic rendering, but their Gaussian primitives are mainly optimized through image-space objectives and remain weakly constrained in unobserved regions. We propose \textit{COVScene}, a pose-free semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. Instead of converting Gaussians to voxels only at evaluation time, COVScene lifts the predicted semantic Gaussians inside the training computation graph, so volumetric regularization provides gradients to Gaussian opacity, geometry, and semantic features. The framework combines a semantic-aware Geometry Transformer, multi-task Gaussian decoding, geometric foundation distillation, and occupancy entropy regularization to support novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction within a single representation. Experiments on ScanNet and ScanNet++ show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline without direct voxel-level supervision.

Related papers

• Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy [38.71779538292353]
  We show how to select the most useful Gaussians when their total number must be limited to control memory and rendering.<n>We propose the Semantic Gaussian Allocation Transformer (SAGFormer), which uses Gaussian attributes and local geometric-semantic features to score candidates and select a fixed final set.<n> Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 show that SAGFormer improves occupancy prediction under the evaluated protocols and yields more semantically consistent and better-utilized Gaussian representations.
  arXiv  Detail & Related papers  (2026-07-24T01:52:49Z)
• OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention [80.51557267896938]
  We present OpenGaFF, a novel framework for open-vocabulary 3D scene understanding built upon 3D Gaussian Splatting.<n>At the core of our method is a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance.<n>To further enforce object-level semantic consistency, we introduce a structured codebook that serves as a set of shared semantic primitives.
  arXiv  Detail & Related papers  (2026-05-07T12:10:07Z)
• Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction [72.85652875805387]
  Free-Range Gaussians is a multi-view reconstruction method that predicts non-pixel, non-voxel-aligned 3D Gaussians from as few as four images.<n>Our generative formulation of reconstruction allows the model to be supervised with non-grid-aligned 3D data.
  arXiv  Detail & Related papers  (2026-04-06T17:24:18Z)
• UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images [10.080087958100552]
  We propose UniSem, a unified framework that improves depth accuracy and semantic generalization via two key components.<n>Error-aware Gaussian Dropout (EGD) performs error-guided capacity control by suppressing redundancy-prone Gaussians.<n>Second, we introduce a Mix-training Curriculum (MTC) that blends 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors.
  arXiv  Detail & Related papers  (2026-03-18T09:26:25Z)
• Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding [86.55824709875598]
  We propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches.<n>Unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor to capture fine-grained 3D shape details.<n>We employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations.
  arXiv  Detail & Related papers  (2026-01-05T18:33:50Z)
• OGGSplat: Open Gaussian Growing for Generalizable Reconstruction with Expanded Field-of-View [74.58230239274123]
  We propose OGGSplat, an open Gaussian growing method that expands the field-of-view in generalizable 3D reconstruction.<n>Our key insight is that the semantic attributes of open Gaussians provide strong priors for image extrapolation.<n> OGGSplat also demonstrates promising semantic-aware scene reconstruction capabilities when provided with two view images captured directly from a smartphone camera.
  arXiv  Detail & Related papers  (2025-06-05T16:17:18Z)
• MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields [100.90743697473232]
  Radiance fields represented by 3D Gaussians excel at synthesizing novel views, offering both high training efficiency and fast rendering.<n>Existing methods often incorporate depth priors from dense estimation networks but overlook the inherent multi-view consistency in input images.<n>We propose a view synthesis framework based on 3D Gaussian Splatting, enabling scene reconstruction from sparse views.
  arXiv  Detail & Related papers  (2024-10-15T08:39:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.