GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervision

• URL: http://arxiv.org/abs/2504.11754v1
• Date: Wed, 16 Apr 2025 04:13:53 GMT
• Title: GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervision
• Authors: Zihui Zhang, Yafei Yang, Hongtao Wen, Bo Yang,
• Abstract summary: We study the hard problem of 3D object segmentation in complex point clouds without requiring human labels of 3D scenes for supervision.<n>By relying on the similarity of pretrained 2D features or external signals such as motion to group 3D points as objects, existing unsupervised methods are usually limited to identifying simple objects like cars or their segmented objects are often inferior due to the lack of objectness in pretrained features.
• Score: 7.511342491529451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the hard problem of 3D object segmentation in complex point clouds without requiring human labels of 3D scenes for supervision. By relying on the similarity of pretrained 2D features or external signals such as motion to group 3D points as objects, existing unsupervised methods are usually limited to identifying simple objects like cars or their segmented objects are often inferior due to the lack of objectness in pretrained features. In this paper, we propose a new two-stage pipeline called GrabS. The core concept of our method is to learn generative and discriminative object-centric priors as a foundation from object datasets in the first stage, and then design an embodied agent to learn to discover multiple objects by querying against the pretrained generative priors in the second stage. We extensively evaluate our method on two real-world datasets and a newly created synthetic dataset, demonstrating remarkable segmentation performance, clearly surpassing all existing unsupervised methods.

Related papers

• DORec: Decomposed Object Reconstruction and Segmentation Utilizing 2D Self-Supervised Features [33.04128557550782]
  We propose a Decomposed Object Reconstruction network based on neural implicit representations. Our key idea is to use 2D self-supervised features to create two levels of masks for supervision: a binary mask for foreground regions and a K-cluster mask for semantically similar regions. Experimental results on different datasets show DORec's superiority in segmenting and reconstructing diverse foreground objects from varied backgrounds enabling downstream tasks such as pose estimation.
  arXiv  Detail & Related papers  (2023-10-17T09:21:29Z)
• PatchContrast: Self-Supervised Pre-training for 3D Object Detection [14.493213289990962]
  We introduce PatchContrast, a novel self-supervised point cloud pre-training framework for 3D object detection.<n>We show that our method outperforms existing state-of-the-art models on three commonly-used 3D detection datasets.
  arXiv  Detail & Related papers  (2023-08-14T07:45:54Z)
• 4D Unsupervised Object Discovery [53.561750858325915]
  We propose 4D unsupervised object discovery, jointly discovering objects from 4D data -- 3D point clouds and 2D RGB images with temporal information. We present the first practical approach for this task by proposing a ClusterNet on 3D point clouds, which is jointly optimized with a 2D localization network.
  arXiv  Detail & Related papers  (2022-10-10T16:05:53Z)
• OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds [4.709764624933227]
  We propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation.
  arXiv  Detail & Related papers  (2022-10-10T07:01:08Z)
• PartAfford: Part-level Affordance Discovery from 3D Objects [113.91774531972855]
  We present a new task of part-level affordance discovery (PartAfford) Given only the affordance labels per object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part corresponds to a certain affordance category. We propose a novel learning framework for PartAfford, which discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization.
  arXiv  Detail & Related papers  (2022-02-28T02:58:36Z)
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection [138.2892824662943]
  A promising solution is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. Recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective.
  arXiv  Detail & Related papers  (2021-08-17T17:56:12Z)
• Unsupervised Object Detection with LiDAR Clues [70.73881791310495]
  We present the first practical method for unsupervised object detection with the aid of LiDAR clues. In our approach, candidate object segments based on 3D point clouds are firstly generated. Then, an iterative segment labeling process is conducted to assign segment labels and to train a segment labeling network. The labeling process is carefully designed so as to mitigate the issue of long-tailed and open-ended distribution.
  arXiv  Detail & Related papers  (2020-11-25T18:59:54Z)
• 3D Registration for Self-Occluded Objects in Context [66.41922513553367]
  We introduce the first deep learning framework capable of effectively handling this scenario. Our method consists of an instance segmentation module followed by a pose estimation one. It allows us to perform 3D registration in a one-shot manner, without requiring an expensive iterative procedure.
  arXiv  Detail & Related papers  (2020-11-23T08:05:28Z)
• SESS: Self-Ensembling Semi-Supervised 3D Object Detection [138.80825169240302]
  We propose SESS, a self-ensembling semi-supervised 3D object detection framework. Specifically, we design a thorough perturbation scheme to enhance generalization of the network on unlabeled and new unseen data. Our SESS achieves competitive performance compared to the state-of-the-art fully-supervised method by using only 50% labeled data.
  arXiv  Detail & Related papers  (2019-12-26T08:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.