Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

• URL: http://arxiv.org/abs/2603.21944v1
• Date: Mon, 23 Mar 2026 13:01:14 GMT
• Title: Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection
• Abstract summary: Group3D is a multi-view open-vocabulary 3D detection framework.<n>It integrates semantic constraints directly into the instance construction process.<n>Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection.
• Score: 23.34626996348395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary 3D object detection aims to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, generating class-agnostic fragments and assigning open-vocabulary categories post hoc. While flexible, such decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging. When geometric evidence is view-dependent and incomplete, this geometry-only merging can lead to irreversible association errors, including over-merging of distinct objects or fragmentation of a single instance. We propose Group3D, a multi-view open-vocabulary 3D detection framework that integrates semantic constraints directly into the instance construction process. Group3D maintains a scene-adaptive vocabulary derived from a multimodal large language model (MLLM) and organizes it into semantic compatibility groups that encode plausible cross-view category equivalence. These groups act as merge-time constraints: 3D fragments are associated only when they satisfy both semantic compatibility and geometric consistency. This semantically gated merging mitigates geometry-driven over-merging while absorbing multi-view category variability. Group3D supports both pose-known and pose-free settings, relying only on RGB observations. Experiments on ScanNet and ARKitScenes demonstrate that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.

Related papers

• ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation [36.94429692322632]
  We present ENEAS, a text-promptable method for instance tracking and semantic discovery.<n>We extend the geometrically robust SeC architecture, previously limited to point interactions, with a text-prompting adapter.<n>The verification layer combines high-speed visual embedding matching with conditional VLM refinement, invoking semantic reasoning only for ambiguous candidates.
  arXiv  Detail & Related papers  (2026-09-03T12:25:26Z)
• GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting [46.95070188813926]
  GroupForward reconstructs geometry, appearance, instance structure, and semantics from sparse, unposed, and uncalibrated images.<n> RSRF constructs an instance-grouped 3D scene graph and retrieves candidate instances for a given referring expression.<n> Experiments on semantic reconstruction and referential reasoning demonstrate the effectiveness of our instance-grouped reconstruction and reasoning framework.
  arXiv  Detail & Related papers  (2026-08-18T08:55:52Z)
• Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views [35.7336094120467]
  A 3D scene is understood through its objects, not the primitives that compose them.<n>We propose a framework that decomposes a scene into instance-structured 3D token groups directly from unposed multi-view images.
  arXiv  Detail & Related papers  (2026-06-28T17:21:45Z)
• FAST3DIS: Feed-forward Anchored Scene Transformer for 3D Instance Segmentation [15.271467111162714]
  FAST3DIS is an end-to-end approach that effectively bypasses post-hoc clustering.<n>We introduce a 3D-anchored, query-based Transformer architecture built upon a foundational depth backbone.<n> Experiments on complex indoor 3D datasets demonstrate that our approach achieves competitive segmentation accuracy.
  arXiv  Detail & Related papers  (2026-03-27T00:45:31Z)
• GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models [70.61152292499737]
  We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm.<n>Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision.<n>We propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation.
  arXiv  Detail & Related papers  (2026-03-17T12:43:48Z)
• Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation [91.2768117730855]
  Generalised 3D Referring Expression (3D-GRES) localizes objects in 3D scenes based on natural language, even when descriptions match multiple or zero targets.<n>Existing methods rely solely on sparse point clouds, lacking rich visual semantics for fine-grained descriptions.<n>We propose HCF-RES, a multi-modal framework with two key innovations.
  arXiv  Detail & Related papers  (2026-03-06T13:09:29Z)
• Instance Consistency Regularization for Semi-Supervised 3D Instance Segmentation [50.51125319374404]
  We propose a novel self-training network InsTeacher3D to explore and exploit pure instance knowledge from unlabeled data. Experimental results on multiple large-scale datasets show that the InsTeacher3D significantly outperforms prior state-of-the-art semi-supervised approaches.
  arXiv  Detail & Related papers  (2024-06-24T16:35:58Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• OpenMask3D: Open-Vocabulary 3D Instance Segmentation [84.58747201179654]
  OpenMask3D is a zero-shot approach for open-vocabulary 3D instance segmentation. Our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings.
  arXiv  Detail & Related papers  (2023-06-23T17:36:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.