OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting

• URL: http://arxiv.org/abs/2603.18510v1
• Date: Thu, 19 Mar 2026 05:37:55 GMT
• Title: OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting
• Abstract summary: We propose OnlinePG, a novel system that integrates geometric reconstruction and open-vocabulary perception.<n>Our method achieves better performance among online approaches, while maintaining real-time efficiency.
• Score: 21.764345239108426
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary scene understanding with online panoptic mapping is essential for embodied applications to perceive and interact with environments. However, existing methods are predominantly offline or lack instance-level understanding, limiting their applicability to real-world robotic tasks. In this paper, we propose OnlinePG, a novel and effective system that integrates geometric reconstruction and open-vocabulary perception using 3D Gaussian Splatting in an online setting. Technically, to achieve online panoptic mapping, we employ an efficient local-to-global paradigm with a sliding window. To build local consistency map, we construct a 3D segment clustering graph that jointly leverages geometric and semantic cues, fusing inconsistent segments within sliding window into complete instances. Subsequently, to update the global map, we construct explicit grids with spatial attributes for the local 3D Gaussian map and fuse them into the global map via robust bidirectional bipartite 3D Gaussian instance matching. Finally, we utilize the fused VLM features inside the 3D spatial attribute grids to achieve open-vocabulary scene understanding. Extensive experiments on widely used datasets demonstrate that our method achieves better performance among online approaches, while maintaining real-time efficiency.

Related papers

• 3D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigation [48.641376758484505]
  Vision-language navigation (VLN) requires an agent to traverse complex 3D environments based on natural language instructions.<n>This work proposes a 3D Gaussian Map that represents the environment as a set of differentiable 3D Gaussians.<n>Egocentric Scene Map is constructed online by initializing 3D Gaussians from sparse pseudo-lidar point clouds.
  arXiv  Detail & Related papers  (2026-05-26T03:25:49Z)
• EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding [66.80528512321106]
  EmbodiedSplat is an online feed-forward 3DGS for open-vocabulary scene understanding.<n>Our objectives are 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner, and 2) Highly generalizable to novel scenes with feed-forward design.
  arXiv  Detail & Related papers  (2026-03-04T16:40:41Z)
• EA3D: Online Open-World 3D Object Extraction from Streaming Videos [55.48835711373918]
  We present ExtractAnything3D (EA3D), a unified online framework for open-world 3D object extraction.<n>Given a streaming video, EA3D dynamically interprets each frame using vision-language and 2D vision foundation encoders to extract object-level knowledge.<n>A recurrent joint optimization module directs the model's attention to regions of interest, simultaneously enhancing both geometric reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2025-10-29T03:56:41Z)
• OV-MAP : Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots [18.200635521222267]
  OV-MAP is a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities.<n>We employ a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds.<n>This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models.
  arXiv  Detail & Related papers  (2025-06-13T08:49:23Z)
• PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAM [105.01907579424362]
  PanoSLAM is the first SLAM system to integrate geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation within a unified framework.<n>For the first time, it achieves panoptic 3D reconstruction of open-world environments directly from the RGB-D video.
  arXiv  Detail & Related papers  (2024-12-31T08:58:10Z)
• ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction [62.599588577671796]
  We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality.
  arXiv  Detail & Related papers  (2023-11-29T20:30:18Z)
• Flattening-Net: Deep Regular 2D Representation for 3D Point Cloud Analysis [66.49788145564004]
  We present an unsupervised deep neural architecture called Flattening-Net to represent irregular 3D point clouds of arbitrary geometry and topology. Our methods perform favorably against the current state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-12-17T15:05:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.