Pano3D: Unified 3D Reconstruction and Panoptic Segmentation
Abstract Overview
Pano3D is a unified framework that performs 3D reconstruction and 3D panoptic segmentation directly from unposed RGB image collections. The method builds on feedforward reconstruction models and adds a Mask2Former-style set-based mask decoder that predicts sequence-consistent semantic and instance masks without heuristic clustering or auxiliary 2D models. It is jointly trained with geometric and semantic objectives, with the decoder features adapted from both encoder and cross-view geometry features. The paper evaluates the approach on ScanNet, ScanNet200, and ScanNet++, and also shows that the framework can be attached to both online memory-based and all-to-all attention reconstruction backbones.
Novelty
The main novelty is a single differentiable architecture that unifies feedforward 3D reconstruction and 3D panoptic segmentation by appending a query-based mask decoder to reconstruction backbones and jointly finetuning them. Unlike several recent alternatives, the method avoids inference-time clustering and explicit multi-view consistency heuristics, and it is demonstrated on two different backbone families.
Results
On the mesh-based UNITE evaluation, Pano3D with a MUSt3R backbone reports state-of-the-art 3D semantic segmentation across ScanNet, ScanNet200, and ScanNet++, and improves class-agnostic 3D instance AP over prior RGB-only baselines on ScanNet and ScanNet200. On the voxel-based IGGT evaluation for ScanNet++, Pano3D with a Pi3 backbone attains 52.54 2D mIoU and 29.46 3D mIoU while maintaining reconstruction quality close to the base model. Ablations indicate that joint geometry-plus-semantics training improves instance localization and can preserve or even improve multi-view geometric consistency relative to frozen features.
Key Points
- Pano3D augments pretrained feedforward reconstruction models with a set-based panoptic decoder to predict 3D-consistent semantic classes and instance masks from unposed RGB views.
- The framework is trained jointly on reconstruction and segmentation, with gradient scaling used to protect geometric quality while allowing semantic supervision to refine shared decoder features.
- Experiments and ablations show strong semantic and instance segmentation performance across ScanNet benchmarks, while also revealing limitations in highly cluttered scenes where query-based masks may merge nearby objects.