EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation
Abstract Overview
EPS3D is an end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation from unposed multi-view images. The method predicts a unified set of 3D panoptic Gaussians that jointly encode geometry, appearance, semantic features, and instance features, rather than relying on separately extracted 2D features followed by 3D fusion. Training uses distillation from 2D foundation models together with a semantic-instance mutual enhancement design that refines instance features with semantic guidance and regularizes semantic features using instance structure. The paper evaluates the approach on ScanNet and Replica, and also demonstrates downstream use in robotic manipulation and 3D scene editing.
Novelty
The paper positions EPS3D as the first end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation from images, directly predicting a unified 3D representation instead of using a two-stage pipeline with precomputed per-view semantic features. A second distinctive element is the explicit semantic-instance mutual enhancement mechanism, which couples category-level and object-level cues through Sem2Ins and Ins2Sem modules.
Results
Across both two-view and multi-view settings on ScanNet and Replica, EPS3D reports stronger semantic and instance segmentation results than the listed baselines while remaining efficient. In the 8-view setting, it achieves 0.6169 novel-view semantic mIoU on ScanNet and 0.4833 on Replica, and its panoptic evaluation reaches PQ 0.5304 on ScanNet and 0.3539 on Replica. The paper also reports fast inference on the order of about one second per scene and shows qualitative applications in robotic grasping and 3D scene editing.
Key Points
- EPS3D directly predicts 3D panoptic Gaussians from multi-view RGB images, jointly modeling geometry, appearance, semantics, and instances in a single feed-forward framework.
- The training strategy combines 2D-teacher distillation with a mutual enhancement module where semantic cues refine instance features and instance relationships enforce semantic consistency.
- Experiments on ScanNet and Replica show improved semantic, instance, and panoptic performance over prior baselines, with additional demonstrations in robotic manipulation and 3D scene editing.