Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation

• URL: http://arxiv.org/abs/2403.06401v1
• Date: Mon, 11 Mar 2024 03:24:58 GMT
• Title: Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation
• Authors: Peng Zhang and Ting Wu and Jinsheng Sun and Weiqing Li and Zhiyong Su
• Abstract summary: We present the first interactive framework for point cloud semantic segmentation, named InterPCSeg. We develop an interaction simulation scheme tailored for the interactive point cloud semantic segmentation task. We evaluate our framework on the S3DIS and ScanNet datasets with off-the-shelf segmentation networks.
• Score: 9.832150567595718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing interactive point cloud segmentation approaches primarily focus on the object segmentation, which aim to determine which points belong to the object of interest guided by user interactions. This paper concentrates on an unexplored yet meaningful task, i.e., interactive point cloud semantic segmentation, which assigns high-quality semantic labels to all points in a scene with user corrective clicks. Concretely, we presents the first interactive framework for point cloud semantic segmentation, named InterPCSeg, which seamlessly integrates with off-the-shelf semantic segmentation networks without offline re-training, enabling it to run in an on-the-fly manner. To achieve online refinement, we treat user interactions as sparse training examples during the test-time. To address the instability caused by the sparse supervision, we design a stabilization energy to regulate the test-time training process. For objective and reproducible evaluation, we develop an interaction simulation scheme tailored for the interactive point cloud semantic segmentation task. We evaluate our framework on the S3DIS and ScanNet datasets with off-the-shelf segmentation networks, incorporating interactions from both the proposed interaction simulator and real users. Quantitative and qualitative experimental results demonstrate the efficacy of our framework in refining the semantic segmentation results with user interactions. The source code will be publicly available.

Related papers

• Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking [59.87033229815062]
  Articulated object manipulation requires precise object interaction, where the object's axis must be carefully considered. Previous research employed interactive perception for manipulating articulated objects, but typically, open-loop approaches often suffer from overlooking the interaction dynamics. We present a closed-loop pipeline integrating interactive perception with online axis estimation from segmented 3D point clouds.
  arXiv  Detail & Related papers  (2024-09-24T17:59:56Z)
• TETRIS: Towards Exploring the Robustness of Interactive Segmentation [39.1981941213761]
  We propose a methodology for finding extreme user inputs by a direct optimization in a white-box adversarial attack on the interactive segmentation model. We report the results of an extensive evaluation of dozens of models.
  arXiv  Detail & Related papers  (2024-02-09T01:36:21Z)
• Interactive segmentation in aerial images: a new benchmark and an open access web-based tool [2.729446374377189]
  In recent years, interactive semantic segmentation proposed in computer vision has achieved an ideal state of human-computer interaction segmentation. This study aims to bridge the gap between interactive segmentation and remote sensing analysis by conducting benchmark study on various interactive segmentation models.
  arXiv  Detail & Related papers  (2023-08-25T04:49:49Z)
• Adaptive Edge-to-Edge Interaction Learning for Point Cloud Analysis [118.30840667784206]
  Key issue for point cloud data processing is extracting useful information from local regions. Previous works ignore the relation between edges in local regions, which encodes the local shape information. This paper proposes a novel Adaptive Edge-to-Edge Interaction Learning module.
  arXiv  Detail & Related papers  (2022-11-20T07:10:14Z)
• Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding [95.78002228538841]
  We propose a new open-world semantic segmentation pipeline that makes the first attempt to learn to segment semantic objects of various open-world categories without any efforts on dense annotations. Our method can directly segment objects of arbitrary categories, outperforming zero-shot segmentation methods that require data labeling on three benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-18T09:20:04Z)
• Multi-Stage Fusion for One-Click Segmentation [20.00726292545008]
  We propose a new multi-stage guidance framework for interactive segmentation. Our proposed framework has a negligible increase in parameter count compared to early-fusion frameworks.
  arXiv  Detail & Related papers  (2020-10-19T17:07:40Z)
• Mixup-CAM: Weakly-supervised Semantic Segmentation via Uncertainty Regularization [73.03956876752868]
  We propose a principled and end-to-end train-able framework to allow the network to pay attention to other parts of the object. Specifically, we introduce the mixup data augmentation scheme into the classification network and design two uncertainty regularization terms to better interact with the mixup strategy.
  arXiv  Detail & Related papers  (2020-08-03T21:19:08Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)
• Bi-Directional Attention for Joint Instance and Semantic Segmentation in Point Clouds [9.434847591440485]
  We build a Bi-Directional Attention module on backbone neural networks for 3D point cloud perception. It uses similarity matrix measured from features for one task to help aggregate non-local information for the other task. From comprehensive experiments and ablation studies on the S3DIS dataset and the PartNet dataset, the superiority of our method is verified.
  arXiv  Detail & Related papers  (2020-03-11T17:16:07Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.