OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting

• URL: http://arxiv.org/abs/2506.07697v1
• Date: Mon, 09 Jun 2025 12:37:15 GMT
• Title: OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting
• Authors: Jens Piekenbrinck, Christian Schmidt, Alexander Hermans, Narunas Vaskevicius, Timm Linder, Bastian Leibe,
• Abstract summary: 3D Gaussian Splatting (3DGS) has emerged as a powerful representation for neural scene reconstruction.<n>We introduce an approach for open-vocabulary 3D instance segmentation without requiring manual labeling, termed OpenSplat3D.<n>We show results on LERF-mask and LERF-OVS as well as the full ScanNet++ validation set, demonstrating the effectiveness of our approach.
• Score: 52.40697058096931
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful representation for neural scene reconstruction, offering high-quality novel view synthesis while maintaining computational efficiency. In this paper, we extend the capabilities of 3DGS beyond pure scene representation by introducing an approach for open-vocabulary 3D instance segmentation without requiring manual labeling, termed OpenSplat3D. Our method leverages feature-splatting techniques to associate semantic information with individual Gaussians, enabling fine-grained scene understanding. We incorporate Segment Anything Model instance masks with a contrastive loss formulation as guidance for the instance features to achieve accurate instance-level segmentation. Furthermore, we utilize language embeddings of a vision-language model, allowing for flexible, text-driven instance identification. This combination enables our system to identify and segment arbitrary objects in 3D scenes based on natural language descriptions. We show results on LERF-mask and LERF-OVS as well as the full ScanNet++ validation set, demonstrating the effectiveness of our approach.

Related papers

• OpenGS-Fusion: Open-Vocabulary Dense Mapping with Hybrid 3D Gaussian Splatting for Refined Object-Level Understanding [17.524454394142477]
  We present OpenGS-Fusion, an innovative open-vocabulary dense mapping framework that improves semantic modeling and refines object-level understanding.<n>We also introduce a novel multimodal language-guided approach named MLLM-Assisted Adaptive Thresholding, which refines the segmentation of 3D objects by adaptively adjusting similarity thresholds.<n>Our method outperforms existing methods in 3D object understanding and scene reconstruction quality, as well as showcasing its effectiveness in language-guided scene interaction.
  arXiv  Detail & Related papers  (2025-08-02T02:22:36Z)
• Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration [41.046653227409564]
  Dr. Splat is a novel approach for open-vocabulary 3D scene understanding leveraging 3D Gaussian Splatting.<n>Our method associates language-aligned CLIP embeddings with 3D Gaussians for holistic 3D scene understanding.<n> Experiments demonstrate that our approach significantly outperforms existing approaches in 3D perception benchmarks.
  arXiv  Detail & Related papers  (2025-02-23T17:01:14Z)
• OVGaussian: Generalizable 3D Gaussian Segmentation with Open Vocabularies [112.80292725951921]
  textbfOVGaussian is a generalizable textbfOpen-textbfVocabulary 3D semantic segmentation framework based on the 3D textbfGaussian representation.<n>We first construct a large-scale 3D scene dataset based on 3DGS, dubbed textbfSegGaussian, which provides detailed semantic and instance annotations for both Gaussian points and multi-view images.<n>To promote semantic generalization across scenes, we introduce Generalizable Semantic Rasterization (GSR), which leverages a
  arXiv  Detail & Related papers  (2024-12-31T07:55:35Z)
• DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting [0.0]
  We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations.<n>We evaluate on synthetic and real-world indoor datasets, demonstrating improved performance over comparable NeRF-based pipelines.
  arXiv  Detail & Related papers  (2024-12-14T21:26:44Z)
• Occam's LGS: An Efficient Approach for Language Gaussian Splatting [57.00354758206751]
  We show that the complicated pipelines for language 3D Gaussian Splatting are simply unnecessary.<n>We apply Occam's razor to the task at hand, leading to a highly efficient weighted multi-view feature aggregation technique.
  arXiv  Detail & Related papers  (2024-12-02T18:50:37Z)
• Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding [59.51535163599723]
  FreeGS is an unsupervised semantic-embedded 3DGS framework that achieves view-consistent 3D scene understanding without the need for 2D labels.<n>FreeGS performs comparably to state-of-the-art methods while avoiding the complex data preprocessing workload.
  arXiv  Detail & Related papers  (2024-11-29T08:52:32Z)
• Search3D: Hierarchical Open-Vocabulary 3D Segmentation [78.47704793095669]
  We introduce Search3D, an approach to construct hierarchical open-vocabulary 3D scene representations.<n>Unlike prior methods, Search3D shifts towards a more flexible open-vocabulary 3D search paradigm.<n>For systematic evaluation, we contribute a scene-scale open-vocabulary 3D part segmentation benchmark based on MultiScan.
  arXiv  Detail & Related papers  (2024-09-27T03:44:07Z)
• AutoInst: Automatic Instance-Based Segmentation of LiDAR 3D Scans [41.17467024268349]
  Making sense of 3D environments requires fine-grained scene understanding. We propose to predict instance segmentations for 3D scenes in an unsupervised way. Our approach attains 13.3% higher Average Precision and 9.1% higher F1 score compared to the best-performing baseline.
  arXiv  Detail & Related papers  (2024-03-24T22:53:16Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding [57.47315482494805]
  Open-world instance-level scene understanding aims to locate and recognize unseen object categories that are not present in the annotated dataset. This task is challenging because the model needs to both localize novel 3D objects and infer their semantic categories. We propose to harness pre-trained vision-language (VL) foundation models that encode extensive knowledge from image-text pairs to generate captions for 3D scenes.
  arXiv  Detail & Related papers  (2023-08-01T07:50:14Z)
• OpenScene: 3D Scene Understanding with Open Vocabularies [73.1411930820683]
  Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries.
  arXiv  Detail & Related papers  (2022-11-28T18:58:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.