TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting

• URL: http://arxiv.org/abs/2605.26576v1
• Date: Tue, 26 May 2026 05:49:12 GMT
• Title: TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting
• Abstract summary: We present TrackRef3D, a fully automatic pipeline that achieves open-world referring segmentation in 3D Gaussian Splatting (3DGS) without manual annotation.<n>We propose a Trajectory-Aware Semantic Consensus Module (TSCM) which aggregates cross-view predictions via synonymous clustering and trajectory-aware voting to establish a canonical semantic identity.
• Score: 10.787720313658093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Referring 3D Gaussian Splatting (R3DGS), which utilizes natural language for 3D object segmentation, has emerged as a crucial capability for embodied AI. However, existing methods typically rely on expensive per-scene manual annotation and per-view pseudo mask generation, which suffer from multi-view inconsistency and poor generalization to varying query specificities. To address this, we present TrackRef3D, a fully automatic pipeline that achieves open-world referring segmentation in 3D Gaussian Splatting (3DGS) without manual annotation by introducing a multi-view consistent track-then-label paradigm that fundamentally decouples object discovery from semantic grounding. Specifically, we propose a Trajectory-Aware Semantic Consensus Module (TSCM) which aggregates cross-view predictions via synonymous clustering and trajectory-aware voting to establish a canonical semantic identity, thereby ensuring multi-view consistency. Furthermore, we employ a visibility-aware description generation strategy to mitigate ambiguity and propose a Hybrid Training Strategy (HTS) that jointly optimizes coarse category semantics and fine-grained referential cues to ensure robustness under varying query specificities using a multi-positive contrastive objective. Extensive experiments on benchmarks demonstrate that TrackRef3D achieves state-of-the-art performance.

Related papers

• NG-GS: NeRF-Guided 3D Gaussian Splatting Segmentation [65.34304674634713]
  We introduce NG-GS, a framework for high-quality object segmentation in 3DGS that explicitly addresses boundary discretization.<n>We show that our method achieves photorealistic state-of-the-art performance, with significant gains in boundary mIoU.
  arXiv  Detail & Related papers  (2026-04-16T07:14:07Z)
• FAST3DIS: Feed-forward Anchored Scene Transformer for 3D Instance Segmentation [15.271467111162714]
  FAST3DIS is an end-to-end approach that effectively bypasses post-hoc clustering.<n>We introduce a 3D-anchored, query-based Transformer architecture built upon a foundational depth backbone.<n> Experiments on complex indoor 3D datasets demonstrate that our approach achieves competitive segmentation accuracy.
  arXiv  Detail & Related papers  (2026-03-27T00:45:31Z)
• SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation [114.57192386025373]
  SegSplat is a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding.<n>This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments.
  arXiv  Detail & Related papers  (2025-11-23T10:26:38Z)
• Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge [45.19482892758984]
  Affordance segmentation aims to parse 3D objects into functionally distinct parts, bridging recognition and interaction for applications in robotic manipulation, embodied AI, and AR.<n>We introduce Cross-Modal Affinity Transfer (CMAT), a pre-training strategy that aligns a 3D encoder with lifted 2D semantics and jointly optimize reconstruction, affinity, and diversity to yield semantically organized representations.<n>We further design the Cross-modal Affordance Transformer (CAST), which integrates multi-modal prompts with CMAT-pretrained features to generate precise, prompt-aware segmentation maps.
  arXiv  Detail & Related papers  (2025-10-09T15:01:26Z)
• Hierarchical Scoring with 3D Gaussian Splatting for Instance Image-Goal Navigation [27.040017548286812]
  Instance Image-Goal Navigation (IIN) requires autonomous agents to identify and navigate to a target object or location depicted in a reference image captured from any viewpoint.<n>We introduce a novel IIN framework with a hierarchical scoring paradigm that estimates optimal viewpoints for target matching.
  arXiv  Detail & Related papers  (2025-06-09T00:58:14Z)
• econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians [56.85804719947]
  We propose econSG for open-vocabulary semantic segmentation with 3DGS.<n>Our econSG shows state-of-the-art performance on four benchmark datasets compared to the existing methods.
  arXiv  Detail & Related papers  (2025-04-08T13:12:31Z)
• Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding [59.51535163599723]
  FreeGS is an unsupervised semantic-embedded 3DGS framework that achieves view-consistent 3D scene understanding without the need for 2D labels.<n>FreeGS performs comparably to state-of-the-art methods while avoiding the complex data preprocessing workload.
  arXiv  Detail & Related papers  (2024-11-29T08:52:32Z)
• InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception [17.530797215534456]
  3D scene understanding has become an essential area of research with applications in autonomous driving, robotics, and augmented reality.<n>We propose InstanceGaussian, a method that jointly learns appearance and semantic features while adaptively aggregating instances.<n>Our approach achieves state-of-the-art performance in category-agnostic, open-vocabulary 3D point-level segmentation.
  arXiv  Detail & Related papers  (2024-11-28T16:08:36Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection [102.62963605429508]
  Point cloud semantic segmentation plays an essential role in autonomous driving. Current 3D semantic segmentation networks focus on convolutional architectures that perform great for well represented classes. We propose a novel Aware 3D Semantic Detection (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task.
  arXiv  Detail & Related papers  (2020-09-22T14:17:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.