Segment and Select: Vision-Language Segmentation in 3D Scenarios

• URL: http://arxiv.org/abs/2606.10594v1
• Date: Tue, 09 Jun 2026 08:58:59 GMT
• Title: Segment and Select: Vision-Language Segmentation in 3D Scenarios
• Abstract summary: 3D vision-language segmentation aims to segment target objects in 3D scenarios according to the linguistic instructions and visual observations.<n>Prior art heavily relies on the coarse superpoint representation to reduce the complexity.<n>We propose the SEGment-And-select (SEGA3D) paradigm for 3D vision-language segmentation that directly operates on the fine-grained visual information.
• Score: 33.55322801240067
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D vision-language segmentation aims to segment target objects in 3D scenarios according to the linguistic instructions and visual observations. Prior art heavily relies on the coarse superpoint representation to reduce the computation complexity, which suffers from poor segmentation quality and messy object boundaries. In this paper, we propose the SEGment-And-select (SEGA3D) paradigm for 3D visionlanguage segmentation that directly operates on the fine-grained visual information and is free from the superpoint dependency. Specifically, we first leverage a mask candidate generator to provide fine-grained categorical mask candidates, substantially improving the quality of candidate masks over the superpoint counterparts. Then, a Large Language Model (LLM) is utilized to generate the semantic and spatial information based on the linguistic description and visual features. The LLM output and visual features are fed to the Semantic-Spatial Selector (SSS) to produce the top-ranking mask candidates. Eventually, the Loopback Verification Module (LVM) is designed to yield the segmentation mask from the selected candidate masks. Our SEGA3D attains competitive performance on ScanRefer, ScanNet and Matterport3D benchmarks. Notably, our SEGA3D surpasses the top-performing counterpart by 8.3 mIoU and 5.3 mIoU on ScanNet and Matterport3D, respectively. Codes will be available upon publication.

Related papers

• OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model [0.0]
  This paper introduces OpenMaskDINO3D, a framework for comprehensive 3D understanding and segmentation.<n>OpenMaskDINO3D processes point cloud data and text prompts to produce instance segmentation masks, excelling in many 3D tasks.
  arXiv  Detail & Related papers  (2025-06-05T09:57:43Z)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation [91.94869042117621]
  Reasoning segmentation aims to segment target objects in complex scenes based on human intent and spatial reasoning.<n>Recent multimodal large language models (MLLMs) have demonstrated impressive 2D image reasoning segmentation.<n>We introduce MLLM-For3D, a framework that transfers knowledge from 2D MLLMs to 3D scene understanding.
  arXiv  Detail & Related papers  (2025-03-23T16:40:20Z)
• Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking [6.599971425078935]
  Existing 3D instance segmentation methods frequently encounter issues with over-segmentation, leading to redundant and inaccurate 3D proposals that complicate downstream tasks. This challenge arises from their unsupervised merging approach, where dense 2D masks are lifted across frames into point clouds to form 3D candidate proposals without direct supervision. We propose a 3D-Aware 2D Mask Tracking module that uses robust 3D priors from a 2D mask segmentation and tracking foundation model (SAM-2) to ensure consistent object masks across video frames.
  arXiv  Detail & Related papers  (2024-11-25T08:26:31Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• 3D-GRES: Generalized 3D Referring Expression Segmentation [77.10044505645064]
  3D Referring Expression (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description. Generalized 3D Referring Expression (3D-GRES) extends the capability to segment any number of instances based on natural language instructions.
  arXiv  Detail & Related papers  (2024-07-30T08:59:05Z)
• RefMask3D: Language-Guided Transformer for 3D Referring Segmentation [32.11635464720755]
  RefMask3D aims to explore the comprehensive multi-modal feature interaction and understanding. RefMask3D outperforms previous state-of-the-art method by a large margin of 3.16% mIoU on the challenging ScanRefer dataset.
  arXiv  Detail & Related papers  (2024-07-25T17:58:03Z)
• Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model [108.35777542298224]
  Reason3D processes point cloud data and text prompts to produce textual responses and segmentation masks.<n>We propose a hierarchical mask decoder that employs a coarse-to-fine approach to segment objects within expansive scenes.
  arXiv  Detail & Related papers  (2024-05-27T17:59:41Z)
• Segment Any 3D Object with Language [58.471327490684295]
  We introduce Segment any 3D Object with LanguagE (SOLE), a semantic geometric and-aware visual-language learning framework with strong generalizability. Specifically, we propose a multimodal fusion network to incorporate multimodal semantics in both backbone and decoder. Our SOLE outperforms previous methods by a large margin on ScanNetv2, ScanNet200, and Replica benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T17:59:10Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.