TSDASeg: A Two-Stage Model with Direct Alignment for Interactive Point Cloud Segmentation

• URL: http://arxiv.org/abs/2506.20991v1
• Date: Thu, 26 Jun 2025 04:10:33 GMT
• Title: TSDASeg: A Two-Stage Model with Direct Alignment for Interactive Point Cloud Segmentation
• Authors: Chade Li, Pengju Zhang, Yihong Wu,
• Abstract summary: We propose TSDASeg, a Two-Stage model coupled with a Direct cross-modal alignment module and memory module for interactive point cloud.<n>We introduce the direct cross-modal alignment module to establish explicit alignment between 3D point clouds and textual/2D image data.<n>Within the memory module, we employ multiple dedicated memory banks to separately store text features, visual features, and their cross-modal correspondence mappings.
• Score: 3.615396917221689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid advancement of 3D vision-language models (VLMs) has spurred significant interest in interactive point cloud processing tasks, particularly for real-world applications. However, existing methods often underperform in point-level tasks, such as segmentation, due to missing direct 3D-text alignment, limiting their ability to link local 3D features with textual context. To solve this problem, we propose TSDASeg, a Two-Stage model coupled with a Direct cross-modal Alignment module and memory module for interactive point cloud Segmentation. We introduce the direct cross-modal alignment module to establish explicit alignment between 3D point clouds and textual/2D image data. Within the memory module, we employ multiple dedicated memory banks to separately store text features, visual features, and their cross-modal correspondence mappings. These memory banks are dynamically leveraged through self-attention and cross-attention mechanisms to update scene-specific features based on prior stored data, effectively addressing inconsistencies in interactive segmentation results across diverse scenarios. Experiments conducted on multiple 3D instruction, reference, and semantic segmentation datasets demonstrate that the proposed method achieves state-of-the-art performance.

Related papers

• PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum [20.206273757144547]
  PGOV3D is a novel framework that introduces a Partial-to-Global curriculum for improving open-vocabulary 3D semantic segmentation.<n>We pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry.<n>In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex.
  arXiv  Detail & Related papers  (2025-06-30T08:13:07Z)
• Relation3D: Enhancing Relation Modeling for Point Cloud Instance Segmentation [4.476845464695504]
  3D instance segmentation aims to predict a set of object instances in a scene, representing them as binary foreground masks with corresponding semantic labels.<n>We introduce textbfRelation3D: Enhancing Relation Modeling for Point Instance. Specifically, we introduce an adaptive superpoint aggregation module and a contrastive learning-guided superpoint refinement module to better represent superpoint features (scene features)<n>Our relation-aware self-attention mechanism enhances the capabilities of modeling relationships between queries by incorporating positional and geometric relationships into the self-attention mechanism.
  arXiv  Detail & Related papers  (2025-06-22T03:48:19Z)
• Unified Representation Space for 3D Visual Grounding [18.652577474202015]
  3D visual grounding aims to identify objects in 3D scenes based on text descriptions.<n>Existing methods rely on separately pre-trained vision and text encoders, resulting in a significant gap between the two modalities.<n>The paper proposes UniSpace-3D, which innovatively introduces a unified representation space for 3DVG.
  arXiv  Detail & Related papers  (2025-06-17T06:53:15Z)
• EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splatting [108.15136508964011]
  EgoSplat is a language-embedded 3D Gaussian Splatting framework for open-vocabulary egocentric scene understanding.<n>EgoSplat achieves state-of-the-art performance in both localization and segmentation tasks on two datasets.
  arXiv  Detail & Related papers  (2025-03-14T12:21:26Z)
• CrossOver: 3D Scene Cross-Modal Alignment [78.3057713547313]
  CrossOver is a novel framework for cross-modal 3D scene understanding.<n>It learns a unified, modality-agnostic embedding space for scenes by aligning modalities.<n>It supports robust scene retrieval and object localization, even with missing modalities.
  arXiv  Detail & Related papers  (2025-02-20T20:05:30Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes [82.4186966781934]
  We introduce a simple, efficient, and effective two-stage detector, termed as Ret3D. At the core of Ret3D is the utilization of novel intra-frame and inter-frame relation modules. With negligible extra overhead, Ret3D achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-18T03:48:58Z)
• Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation [90.74732705236336]
  Language-queried video actor segmentation aims to predict the pixel-mask of the actor which performs the actions described by a natural language query in the target frames. We propose a collaborative spatial-temporal encoder-decoder framework which contains a 3D temporal encoder over the video clip to recognize the queried actions, and a 2D spatial encoder over the target frame to accurately segment the queried actors.
  arXiv  Detail & Related papers  (2021-05-14T13:27:53Z)
• LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment [0.0]
  We propose a recurrent segmentation architecture (RNN), which takes a single range image frame as input. An alignment strategy, which we call Temporal Memory Alignment, uses ego motion to temporally align the memory between consecutive frames in feature space. We demonstrate the benefits of the presented approach on two large-scale datasets and compare it to several stateof-the-art methods.
  arXiv  Detail & Related papers  (2021-03-03T09:01:45Z)
• Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes [36.07733308424772]
  The deficiency of 3D segmentation labels is one of the main obstacles to effective point cloud segmentation. We propose a novel deep graph convolutional network-based framework for large-scale semantic scene segmentation in point clouds with sole 2D supervision.
  arXiv  Detail & Related papers  (2020-04-26T23:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.