ClickSeg3D: Few-Click Interactive Segmentation via Semantic Embeddings

• URL: http://arxiv.org/abs/2605.08925v2
• Date: Sat, 16 May 2026 09:07:00 GMT
• Title: ClickSeg3D: Few-Click Interactive Segmentation via Semantic Embeddings
• Abstract summary: We propose a novel interactive segmentation framework that operates directly on sparse, randomly downsampled 3D points.<n>Our framework consists of a point Transformer-based encoder and a hierarchical mask decoder, which integrates multi-level crop-and-merge operations.<n>Our model improves the mIoU metric by over 20 percent compared to strong baselines.
• Score: 9.399068602086821
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interactive segmentation allows efficient label generation by leveraging user-provided clicks to progressively refine predictions, which is critical when fully supervised labels are costly or generalization to unseen classes is needed. Existing 3D interactive methods are limited: most operate sequentially, predicting only one object per iteration with binary masks, while several recent approaches depend on 2D foundation models and camera alignment to bridge the 2D-3D gap. To address these limitations, we propose a novel interactive segmentation framework that operates directly on sparse, randomly downsampled 3D points and processes multiple object clicks in a single forward pass. Our framework consists of a point Transformer-based encoder and a hierarchical mask decoder, which integrates multi-level crop-and-merge operations conditioned on learnable semantic embeddings. Unlike prior interactive approaches that require repeated model updates after each manually corrective click, our method jointly reasons over all click queries, modeling inter-instance relationships and refining both spatial masks and semantic predictions through spatial and semantic embeddings. Extensive experiments demonstrate that our model improves the mIoU metric by over 20 percent compared to strong baselines and achieves 8-10 percent gains under cross-dataset evaluation for a one-click per instance setting, often requiring only a single click per object. Our approach provides a generalizable and efficient solution for interactive 3D instance segmentation, particularly suitable for real-time applications such as robotic manipulation, navigation, and rapid 3D semantic annotation.

Related papers

• ISRS-DETR: Detection-Guided Click Propagation for Remote Sensing Interactive Segmentation [9.07529044229261]
  We propose ISRS-DETR, a detection-guided interactive segmentation framework.<n>It injects object-level evidence into both training and inference.<n>It achieves state-of-the-art accuracy while substantially reducing Number Clicks per Image.
  arXiv  Detail & Related papers  (2026-08-03T16:37:12Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• REACT3D: Recovering Articulations for Interactive Physical 3D Scenes [96.27769519526426]
  REACT3D is a framework that converts static 3D scenes into simulation-ready interactive replicas with consistent geometry.<n>We achieve state-of-the-art performance on detection/segmentation and articulation metrics across diverse indoor scenes.
  arXiv  Detail & Related papers  (2025-10-13T12:37:59Z)
• Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes [71.2827490406779]
  We propose NPISeg3D, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges.<n>NPISeg3D introduces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization.<n>We design a prototype modulator that adaptively modulates click prototypes with object-specific latent variables, improving the model's ability to capture object-aware context.
  arXiv  Detail & Related papers  (2025-05-03T07:43:23Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Articulated Object Manipulation using Online Axis Estimation with SAM2-Based Tracking [57.942404069484134]
  Articulated object manipulation requires precise object interaction, where the object's axis must be carefully considered.<n>Previous research employed interactive perception for manipulating articulated objects, but typically, open-loop approaches often suffer from overlooking the interaction dynamics.<n>We present a closed-loop pipeline integrating interactive perception with online axis estimation from segmented 3D point clouds.
  arXiv  Detail & Related papers  (2024-09-24T17:59:56Z)
• iDet3D: Towards Efficient Interactive Object Detection for LiDAR Point Clouds [39.261055567560724]
  We propose iDet3D, an efficient interactive 3D object detector. iDet3D supports a user-friendly 2D interface, which can ease the cognitive burden of exploring 3D space. We show that our method can construct precise annotations in a few clicks.
  arXiv  Detail & Related papers  (2023-12-24T09:59:46Z)
• Multi-label affordance mapping from egocentric vision [3.683202928838613]
  We present a new approach to affordance perception which enables accurate multi-label segmentation. Our approach can be used to automatically extract grounded affordances from first person videos. We show how our metric representation can be exploited for build a map of interaction hotspots.
  arXiv  Detail & Related papers  (2023-09-05T10:56:23Z)
• AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation [32.63772366307106]
  We introduce AGILE3D, an efficient, attention-based model that supports simultaneous segmentation of multiple 3D objects. Our core idea is to encode user clicks as spatial-temporal queries and enable explicit interactions between click queries and the 3D scene. In experiments with four different 3D point cloud datasets, AGILE3D sets a new state-of-the-art.
  arXiv  Detail & Related papers  (2023-06-01T17:59:10Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• PointInst3D: Segmenting 3D Instances by Points [136.7261709896713]
  We propose a fully-convolutional 3D point cloud instance segmentation method that works in a per-point prediction fashion. We find the key to its success is assigning a suitable target to each sampled point. Our approach achieves promising results on both ScanNet and S3DIS benchmarks.
  arXiv  Detail & Related papers  (2022-04-25T02:41:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.