DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting

• URL: http://arxiv.org/abs/2412.10972v1
• Date: Sat, 14 Dec 2024 21:26:44 GMT
• Title: DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting
• Authors: Luis Wiedmann, Luca Wiehe, David Rozenberszki,
• Abstract summary: Open-set 3D segmentation represents a major point of interest for downstream robotics and augmented/virtual reality applications.<n>We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations and semantic segmentation foundation models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. Recent advances introduce 3D Gaussian Splatting as a computationally efficient representation of the underlying scene. They enable the rendering of novel views while achieving real-time display rates and matching the quality of computationally far more expensive methods. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations and semantic segmentation foundation models. The pipeline proposes class-agnostic masks based on a 3D reconstruction of the scene. Given the resulting class-agnostic masks, we use a class-aware 2D foundation model to add class annotations to the 3D masks. We test this pipeline with 3D Gaussian Splatting and different 2D segmentation models and achieve better performance than more tailored approaches while also significantly increasing the modularity.

Related papers

• Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding [59.51535163599723]
  FreeGS is an unsupervised semantic-embedded 3DGS framework that achieves view-consistent 3D scene understanding without the need for 2D labels. We show that FreeGS performs comparably to state-of-the-art methods while avoiding the complex data preprocessing workload.
  arXiv  Detail & Related papers  (2024-11-29T08:52:32Z)
• XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation [72.12250272218792]
  We propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. We integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks. The generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings.
  arXiv  Detail & Related papers  (2024-11-20T12:02:12Z)
• Zero-Shot Dual-Path Integration Framework for Open-Vocabulary 3D Instance Segmentation [19.2297264550686]
  Open-vocabulary 3D instance segmentation transcends traditional closed-vocabulary methods. We introduce Zero-Shot Dual-Path Integration Framework that equally values the contributions of both 3D and 2D modalities. Our framework, utilizing pre-trained models in a zero-shot manner, is model-agnostic and demonstrates superior performance on both seen and unseen data.
  arXiv  Detail & Related papers  (2024-08-16T07:52:00Z)
• 3x2: 3D Object Part Segmentation by 2D Semantic Correspondences [33.99493183183571]
  We propose to leverage a few annotated 3D shapes or richly annotated 2D datasets to perform 3D object part segmentation. We present our novel approach, termed 3-By-2 that achieves SOTA performance on different benchmarks with various granularity levels.
  arXiv  Detail & Related papers  (2024-07-12T19:08:00Z)
• Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models [20.277479473218513]
  We introduce a new task: Zero-Shot 3D Reasoning for parts searching and localization for objects. We design a simple baseline method, Reasoning3D, with the capability to understand and execute complex commands. We show that Reasoning3D can effectively localize and highlight parts of 3D objects based on implicit textual queries.
  arXiv  Detail & Related papers  (2024-05-29T17:56:07Z)
• Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting [27.974762304763694]
  We introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features into a novel semantic component of 3D Gaussians. We build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference.
  arXiv  Detail & Related papers  (2024-03-22T21:28:19Z)
• Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels [141.23836433191624]
  Current 3D scene segmentation methods are heavily dependent on manually annotated 3D training datasets. We propose Segment3D, a method for class-agnostic 3D scene segmentation that produces high-quality 3D segmentation masks.
  arXiv  Detail & Related papers  (2023-12-28T18:57:11Z)
• 2D-Guided 3D Gaussian Segmentation [15.139488857163064]
  This paper introduces a 3D Gaussian segmentation method implemented with 2D segmentation as supervision. This approach uses input 2D segmentation maps to guide the learning of the added 3D Gaussian semantic information. Experiments show that our method can achieve comparable performances on mIOU and mAcc for multi-object segmentation.
  arXiv  Detail & Related papers  (2023-12-26T13:28:21Z)
• SAM-guided Graph Cut for 3D Instance Segmentation [60.75119991853605]
  This paper addresses the challenge of 3D instance segmentation by simultaneously leveraging 3D geometric and multi-view image information. We introduce a novel 3D-to-2D query framework to effectively exploit 2D segmentation models for 3D instance segmentation. Our method achieves robust segmentation performance and can generalize across different types of scenes.
  arXiv  Detail & Related papers  (2023-12-13T18:59:58Z)
• Segment Any 3D Gaussians [85.93694310363325]
  This paper presents SAGA, a highly efficient 3D promptable segmentation method based on 3D Gaussian Splatting (3D-GS) Given 2D visual prompts as input, SAGA can segment the corresponding 3D target represented by 3D Gaussians within 4 ms. We show that SAGA achieves real-time multi-granularity segmentation with quality comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-01T17:15:24Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• Semi-supervised 3D shape segmentation with multilevel consistency and part substitution [21.075426681857024]
  We propose an effective semi-supervised method for learning 3D segmentations from a few labeled 3D shapes and a large amount of unlabeled 3D data. For the unlabeled data, we present a novel multilevel consistency loss to enforce consistency of network predictions between perturbed copies of a 3D shape. For the labeled data, we develop a simple yet effective part substitution scheme to augment the labeled 3D shapes with more structural variations to enhance training.
  arXiv  Detail & Related papers  (2022-04-19T11:48:24Z)
• ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation [19.575077449759377]
  We propose ICM-3D, a single-step method to segment 3D instances via instantiated categorization. We conduct extensive experiments to verify the effectiveness of ICM-3D and show that it obtains inspiring performance across multiple frameworks, backbones and benchmarks.
  arXiv  Detail & Related papers  (2021-08-26T13:08:37Z)
• Multi-Modality Task Cascade for 3D Object Detection [22.131228757850373]
  Many methods train two models in isolation and use simple feature concatenation to represent 3D sensor data. We propose a novel Multi-Modality Task Cascade network (MTC-RCNN) that leverages 3D box proposals to improve 2D segmentation predictions. We show that including a 2D network between two stages of 3D modules significantly improves both 2D and 3D task performance.
  arXiv  Detail & Related papers  (2021-07-08T17:55:01Z)
• A Convolutional Architecture for 3D Model Embedding [1.3858051019755282]
  We propose a deep learning architecture to handle 3D models as an input. We show that the embedding representation conveys semantic information that helps to deal with the similarity assessment of 3D objects.
  arXiv  Detail & Related papers  (2021-03-05T15:46:47Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.