FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models

• URL: http://arxiv.org/abs/2603.06166v1
• Date: Fri, 06 Mar 2026 11:19:33 GMT
• Title: FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models
• Abstract summary: FreeOcc is a training-free pipeline that recovers semantics and geometry from multi-view images.<n>On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods.
• Score: 0.9345376836714131
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.

Related papers

• Registration-Free Learnable Multi-View Capture of Faces in Dense Semantic Correspondence [30.38509036690193]
  MOCHI is a multi-view 3D face prediction framework trained without requiring registered training data.<n>Standard point-to-surface distances induce training instabilities and visual artifacts in registration-free settings.<n>We propose pointmap- and normal-based losses instead, which provide smoother gradients and superior reconstruction fidelity.
  arXiv  Detail & Related papers  (2026-05-02T13:59:29Z)
• FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction [11.503430067699442]
  FreeOcc is a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences.<n>FreeOcc operates without 3D annotations, pose ground truth, or any learning stage.<n>FreeOcc achieves over $2times$ improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods.
  arXiv  Detail & Related papers  (2026-04-30T17:05:56Z)
• TAP-CT: 3D Task-Agnostic Pretraining of Computed Tomography Foundation Models [39.00742360251856]
  Existing foundation models (FMs) in the medical domain often require extensive fine-tuning or rely on training resource-intensive decoders.<n>We introduce a suite of task-agnostic pretraining of CT foundation models (TAP-CT)<n>Our approach incorporates targeted modifications to patch embeddings, positional encodings, and volumetric augmentations, making the architecture depth-aware.
  arXiv  Detail & Related papers  (2025-11-30T12:43:15Z)
• ShelfOcc: Native 3D Supervision beyond LiDAR for Vision-Based Occupancy Estimation [9.977834471775816]
  We introduce ShelfOcc, a vision-only method that overcomes limitations without relying on LiDAR.<n> ShelfOcc brings supervision into native 3D space by generating metrically consistent semantic voxel labels from video.<n>Our method introduces a dedicated framework that mitigates these issues by filtering and accumulating static geometry consistently across frames.
  arXiv  Detail & Related papers  (2025-11-19T12:44:13Z)
• Zero-shot Inexact CAD Model Alignment from a Single Image [53.37898107159792]
  A practical approach to infer 3D scene structure from a single image is to retrieve a closely matching 3D model from a database and align it with the object in the image.<n>Existing methods rely on supervised training with images and pose annotations, which limits them to a narrow set of object categories.<n>We propose a weakly supervised 9-DoF alignment method for inexact 3D models that requires no pose annotations and generalizes to unseen categories.
  arXiv  Detail & Related papers  (2025-07-04T04:46:59Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)
• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks [75.10231099007494]
  We introduce a self-supervised pretraining method, called OccFeat, for Bird's-Eye-View (BEV) segmentation networks. With OccFeat, we pretrain a BEV network via occupancy prediction and feature distillation tasks. Models pretrained with our method exhibit improved BEV semantic segmentation performance, particularly in low-data scenarios.
  arXiv  Detail & Related papers  (2024-04-22T09:43:03Z)
• OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environments [77.0399450848749]
  We propose an OccNeRF method for training occupancy networks without 3D supervision. We parameterize the reconstructed occupancy fields and reorganize the sampling strategy to align with the cameras' infinite perceptive range. For semantic occupancy prediction, we design several strategies to polish the prompts and filter the outputs of a pretrained open-vocabulary 2D segmentation model.
  arXiv  Detail & Related papers  (2023-12-14T18:58:52Z)
• Semi-Supervised Single-View 3D Reconstruction via Prototype Shape Priors [79.80916315953374]
  We propose SSP3D, a semi-supervised framework for 3D reconstruction. We introduce an attention-guided prototype shape prior module for guiding realistic object reconstruction. Our approach also performs well when transferring to real-world Pix3D datasets under labeling ratios of 10%.
  arXiv  Detail & Related papers  (2022-09-30T11:19:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.