SA3DIP: Segment Any 3D Instance with Potential 3D Priors

• URL: http://arxiv.org/abs/2411.03819v1
• Date: Wed, 06 Nov 2024 10:39:00 GMT
• Title: SA3DIP: Segment Any 3D Instance with Potential 3D Priors
• Authors: Xi Yang, Xu Gu, Xingyilang Yin, Xinbo Gao,
• Abstract summary: We propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process.
• Score: 41.907914881608995
• License:
• Abstract: The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based on estimated normal from spatial coordinates, resulting in under-segmentation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process. Furthermore, we notice a considerable portion of low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D class-agnostic instance segmentation. Experimental evaluations on various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2-INS dataset are available HERE.

Related papers

• General Geometry-aware Weakly Supervised 3D Object Detection [62.26729317523975]
  A unified framework is developed for learning 3D object detectors from RGB images and associated 2D boxes. Experiments on KITTI and SUN-RGBD datasets demonstrate that our method yields surprisingly high-quality 3D bounding boxes with only 2D annotation.
  arXiv  Detail & Related papers  (2024-07-18T17:52:08Z)
• AutoInst: Automatic Instance-Based Segmentation of LiDAR 3D Scans [41.17467024268349]
  Making sense of 3D environments requires fine-grained scene understanding. We propose to predict instance segmentations for 3D scenes in an unsupervised way. Our approach attains 13.3% higher Average Precision and 9.1% higher F1 score compared to the best-performing baseline.
  arXiv  Detail & Related papers  (2024-03-24T22:53:16Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• SAM-guided Graph Cut for 3D Instance Segmentation [60.75119991853605]
  This paper addresses the challenge of 3D instance segmentation by simultaneously leveraging 3D geometric and multi-view image information. We introduce a novel 3D-to-2D query framework to effectively exploit 2D segmentation models for 3D instance segmentation. Our method achieves robust segmentation performance and can generalize across different types of scenes.
  arXiv  Detail & Related papers  (2023-12-13T18:59:58Z)
• PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimation [32.2861030554128]
  PartSLIP, a recent advancement, has made significant strides in zero- and few-shot 3D part segmentation. We introduce PartSLIP++, an enhanced version designed to overcome the limitations of its predecessor. We show that PartSLIP++ demonstrates better performance over PartSLIP in both low-shot 3D semantic and instance-based object part segmentation tasks.
  arXiv  Detail & Related papers  (2023-12-05T01:33:04Z)
• UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes [35.38074724231105]
  UnScene3D is a fully unsupervised 3D learning approach for class-agnostic 3D instance segmentation of indoor scans. We operate on a basis of geometric oversegmentation, enabling efficient representation and learning on high-resolution 3D data. Our approach improves over state-of-the-art unsupervised 3D instance segmentation methods by more than 300% Average Precision score.
  arXiv  Detail & Related papers  (2023-03-25T19:15:16Z)
• Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception [122.53774221136193]
  State-of-the-art methods for driving-scene LiDAR-based perception often project the point clouds to 2D space and then process them via 2D convolution. A natural remedy is to utilize the 3D voxelization and 3D convolution network. We propose a new framework for the outdoor LiDAR segmentation, where cylindrical partition and asymmetrical 3D convolution networks are designed to explore the 3D geometric pattern.
  arXiv  Detail & Related papers  (2021-09-12T06:25:11Z)
• 3D Guided Weakly Supervised Semantic Segmentation [27.269847900950943]
  We propose a weakly supervised 2D semantic segmentation model by incorporating sparse bounding box labels with available 3D information. We manually labeled a subset of the 2D-3D Semantics(2D-3D-S) dataset with bounding boxes, and introduce our 2D-3D inference module to generate accurate pixel-wise segment proposal masks.
  arXiv  Detail & Related papers  (2020-12-01T03:34:15Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.