DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting

• URL: http://arxiv.org/abs/2412.10972v1
• Date: Sat, 14 Dec 2024 21:26:44 GMT
• Title: DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting
• Authors: Luis Wiedmann, Luca Wiehe, David Rozenberszki,
• Abstract summary: Open-set 3D segmentation represents a major point of interest for downstream robotics and augmented/virtual reality applications. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations and semantic segmentation foundation models.
• Score: 0.0
• License:
• Abstract: Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. Recent advances introduce 3D Gaussian Splatting as a computationally efficient representation of the underlying scene. They enable the rendering of novel views while achieving real-time display rates and matching the quality of computationally far more expensive methods. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations and semantic segmentation foundation models. The pipeline proposes class-agnostic masks based on a 3D reconstruction of the scene. Given the resulting class-agnostic masks, we use a class-aware 2D foundation model to add class annotations to the 3D masks. We test this pipeline with 3D Gaussian Splatting and different 2D segmentation models and achieve better performance than more tailored approaches while also significantly increasing the modularity.

Related papers

• Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting [27.974762304763694]
  We introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features into a novel semantic component of 3D Gaussians. We build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference.
  arXiv  Detail & Related papers  (2024-03-22T21:28:19Z)
• SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition [66.56357905500512]
  3D Gaussian Splatting has emerged as an alternative 3D representation for novel view synthesis. We propose SAGD, a conceptually simple yet effective boundary-enhanced segmentation pipeline for 3D-GS. Our approach achieves high-quality 3D segmentation without rough boundary issues, which can be easily applied to other scene editing tasks.
  arXiv  Detail & Related papers  (2024-01-31T14:19:03Z)
• Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels [141.23836433191624]
  Current 3D scene segmentation methods are heavily dependent on manually annotated 3D training datasets. We propose Segment3D, a method for class-agnostic 3D scene segmentation that produces high-quality 3D segmentation masks.
  arXiv  Detail & Related papers  (2023-12-28T18:57:11Z)
• 2D-Guided 3D Gaussian Segmentation [15.139488857163064]
  This paper introduces a 3D Gaussian segmentation method implemented with 2D segmentation as supervision. This approach uses input 2D segmentation maps to guide the learning of the added 3D Gaussian semantic information. Experiments show that our method can achieve comparable performances on mIOU and mAcc for multi-object segmentation.
  arXiv  Detail & Related papers  (2023-12-26T13:28:21Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• SAM-guided Graph Cut for 3D Instance Segmentation [60.75119991853605]
  This paper addresses the challenge of 3D instance segmentation by simultaneously leveraging 3D geometric and multi-view image information. We introduce a novel 3D-to-2D query framework to effectively exploit 2D segmentation models for 3D instance segmentation. Our method achieves robust segmentation performance and can generalize across different types of scenes.
  arXiv  Detail & Related papers  (2023-12-13T18:59:58Z)
• Segment Any 3D Gaussians [85.93694310363325]
  This paper presents SAGA, a highly efficient 3D promptable segmentation method based on 3D Gaussian Splatting (3D-GS) Given 2D visual prompts as input, SAGA can segment the corresponding 3D target represented by 3D Gaussians within 4 ms. We show that SAGA achieves real-time multi-granularity segmentation with quality comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-01T17:15:24Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation [19.575077449759377]
  We propose ICM-3D, a single-step method to segment 3D instances via instantiated categorization. We conduct extensive experiments to verify the effectiveness of ICM-3D and show that it obtains inspiring performance across multiple frameworks, backbones and benchmarks.
  arXiv  Detail & Related papers  (2021-08-26T13:08:37Z)
• A Convolutional Architecture for 3D Model Embedding [1.3858051019755282]
  We propose a deep learning architecture to handle 3D models as an input. We show that the embedding representation conveys semantic information that helps to deal with the similarity assessment of 3D objects.
  arXiv  Detail & Related papers  (2021-03-05T15:46:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.