GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting

• URL: http://arxiv.org/abs/2608.17535v1
• Date: Tue, 18 Aug 2026 08:55:52 GMT
• Title: GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting
• Abstract summary: GroupForward reconstructs geometry, appearance, instance structure, and semantics from sparse, unposed, and uncalibrated images.<n> RSRF constructs an instance-grouped 3D scene graph and retrieves candidate instances for a given referring expression.<n> Experiments on semantic reconstruction and referential reasoning demonstrate the effectiveness of our instance-grouped reconstruction and reasoning framework.
• Score: 46.95070188813926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simultaneously reconstructing and understanding 3D environments is essential for embodied agents. Toward this goal, feed-forward semantic 3D Gaussian Splatting (3DGS) efficiently constructs semantic scene representations from sparse multi-view observations. However, existing methods lack explicit instance discrimination and mainly support category- or phrase-based semantic queries. To this end, we propose GroupForward, an instance-grouped feed-forward Gaussian splatting model that reconstructs geometry, appearance, instance structure, and semantics from sparse, unposed, and uncalibrated multi-view images. Unlike existing methods that attach high-dimensional semantic features to each Gaussian, GroupForward learns compact instance embeddings that group Gaussians into cross-view consistent 3D instances, reformulating feed-forward semantic 3DGS from per-Gaussian semantic feature rendering to instance-level semantic aggregation and propagation. Building on these instance groups, we further propose a Referential Scene Reasoning Framework (RSRF) for complex 3D referring segmentation. RSRF constructs an instance-grouped 3D scene graph and retrieves candidate instances for a given referring expression. A vision-language model then reasons over structured instance evidence and multi-view observations to identify the referred instance among the candidates. RSRF thereby extends language interaction from simple semantic querying to complex referential scene reasoning. Experiments on semantic reconstruction and referential reasoning demonstrate the effectiveness of our instance-grouped reconstruction and reasoning framework.

Related papers

• USR-Drive: Unified Driving Scene Representation via Joint Denoising of 3D Gaussians and Boxes [57.4583999658488]
  Spatial representation learning for autonomous driving aims to map raw visual signals into structured 3D scene representations.<n>Existing methods treat dynamic reconstruction and instance-level perception as separate tasks.<n>We propose USR-Drive, a unified conditional generative framework that recovers dense dynamic geometry and instance-level object layouts within a shared scene representation.
  arXiv  Detail & Related papers  (2026-08-19T15:29:06Z)
• InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding [7.875166305092363]
  We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding.<n>In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics.
  arXiv  Detail & Related papers  (2026-08-07T12:02:48Z)
• Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors [4.720025219010595]
  GaussDet is a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities.<n>Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding.
  arXiv  Detail & Related papers  (2026-06-29T17:58:41Z)
• FAST3DIS: Feed-forward Anchored Scene Transformer for 3D Instance Segmentation [15.271467111162714]
  FAST3DIS is an end-to-end approach that effectively bypasses post-hoc clustering.<n>We introduce a 3D-anchored, query-based Transformer architecture built upon a foundational depth backbone.<n> Experiments on complex indoor 3D datasets demonstrate that our approach achieves competitive segmentation accuracy.
  arXiv  Detail & Related papers  (2026-03-27T00:45:31Z)
• Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection [23.34626996348395]
  Group3D is a multi-view open-vocabulary 3D detection framework.<n>It integrates semantic constraints directly into the instance construction process.<n>Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection.
  arXiv  Detail & Related papers  (2026-03-23T13:01:14Z)
• SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation [114.57192386025373]
  SegSplat is a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding.<n>This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments.
  arXiv  Detail & Related papers  (2025-11-23T10:26:38Z)
• ReferSplat: Referring Segmentation in 3D Gaussian Splatting [60.73702075842278]
  Referring 3D Gaussian Splatting (R3DGS)<n>Task aims to segment target objects in a 3D Gaussian scene based on natural language descriptions.<n>To address these challenges, we propose ReferSplat, a framework that explicitly models 3D Gaussian points with natural language expressions.
  arXiv  Detail & Related papers  (2025-08-11T17:59:30Z)
• OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting [52.40697058096931]
  3D Gaussian Splatting (3DGS) has emerged as a powerful representation for neural scene reconstruction.<n>We introduce an approach for open-vocabulary 3D instance segmentation without requiring manual labeling, termed OpenSplat3D.<n>We show results on LERF-mask and LERF-OVS as well as the full ScanNet++ validation set, demonstrating the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-06-09T12:37:15Z)
• InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception [17.530797215534456]
  3D scene understanding has become an essential area of research with applications in autonomous driving, robotics, and augmented reality.<n>We propose InstanceGaussian, a method that jointly learns appearance and semantic features while adaptively aggregating instances.<n>Our approach achieves state-of-the-art performance in category-agnostic, open-vocabulary 3D point-level segmentation.
  arXiv  Detail & Related papers  (2024-11-28T16:08:36Z)
• SAI3D: Segment Any Instance in 3D Scenes [68.57002591841034]
  We introduce SAI3D, a novel zero-shot 3D instance segmentation approach. Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations. Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach.
  arXiv  Detail & Related papers  (2023-12-17T09:05:47Z)
• Convolutional Occupancy Networks [88.48287716452002]
  We propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.
  arXiv  Detail & Related papers  (2020-03-10T10:17:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.