FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

• URL: http://arxiv.org/abs/2604.28115v1
• Date: Thu, 30 Apr 2026 17:05:56 GMT
• Title: FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
• Abstract summary: FreeOcc is a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences.<n>FreeOcc operates without 3D annotations, pose ground truth, or any learning stage.<n>FreeOcc achieves over $2times$ improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods.
• Score: 11.503430067699442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over $2\times$ improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.

Related papers

• GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation [55.47130289185285]
  Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation.<n>We present textbfMethodname, a compact, policy-native extension that inserts textbfWorld State Tokens and textbfWorld Prediction Tokens directly into the VLA backbone.
  arXiv  Detail & Related papers  (2026-08-26T11:41:55Z)
• Gaussian-JEPA: Joint-Embedding Predictive Learning for 3D Gaussian Splats [91.55220894541151]
  3D Gaussian Splatting (3DGS) represents 3D content with anisotropic primitives that jointly encode geometry and appearance.<n>We introduce Gaussian-JEPA, which predicts representations of held-out Gaussian token blocks from visible context.
  arXiv  Detail & Related papers  (2026-08-16T09:43:50Z)
• Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images [62.31892851760248]
  COVScene is a pose-free semantic Gaussian framework that couples renderable primitives with a dense semantic occupancy field.<n>We show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline.
  arXiv  Detail & Related papers  (2026-07-02T03:00:23Z)
• FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models [0.9345376836714131]
  FreeOcc is a training-free pipeline that recovers semantics and geometry from multi-view images.<n>On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods.
  arXiv  Detail & Related papers  (2026-03-06T11:19:33Z)
• Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes [12.60608820766928]
  Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments.<n>We propose a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free)<n>Our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU.
  arXiv  Detail & Related papers  (2026-02-26T06:37:43Z)
• TT-Occ: Test-Time Compute for Self-Supervised Occupancy via Spatio-Temporal Gaussian Splatting [32.57885385644153]
  Self-supervised 3D occupancy prediction offers a promising solution for understanding driving scenes without requiring costly 3D annotations.<n>We propose a practical and flexible test-time occupancy prediction framework termed TT-Occ.
  arXiv  Detail & Related papers  (2025-03-11T14:37:39Z)
• GaussRender: Learning 3D Occupancy with Gaussian Rendering [86.89653628311565]
  GaussRender is a module that improves 3D occupancy learning by enforcing projective consistency.<n>Our method penalizes 3D configurations that produce inconsistent 2D projections, thereby enforcing a more coherent 3D structure.
  arXiv  Detail & Related papers  (2025-02-07T16:07:51Z)
• EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding [72.96388875744704]
  3D occupancy prediction provides a comprehensive description of the surrounding scenes.<n>Most existing methods focus on offline perception from one or a few views.<n>We formulate an embodied 3D occupancy prediction task to target this practical scenario and propose a Gaussian-based EmbodiedOcc framework to accomplish it.
  arXiv  Detail & Related papers  (2024-12-05T17:57:09Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)
• OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environments [77.0399450848749]
  We propose an OccNeRF method for training occupancy networks without 3D supervision. We parameterize the reconstructed occupancy fields and reorganize the sampling strategy to align with the cameras' infinite perceptive range. For semantic occupancy prediction, we design several strategies to polish the prompts and filter the outputs of a pretrained open-vocabulary 2D segmentation model.
  arXiv  Detail & Related papers  (2023-12-14T18:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.