SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images

• URL: http://arxiv.org/abs/2603.19926v1
• Date: Fri, 20 Mar 2026 13:10:46 GMT
• Title: SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images
• Abstract summary: SegVGGT is a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation.<n>Our method deeply integrates instance identification into the visual geometry grounded transformer.<n>Experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200.
• Score: 11.617237358347777
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstruction noise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semantic understanding. In this work, we present SegVGGT, a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation directly from multi-view RGB images. By introducing object queries that interact with multi-level geometric features, our method deeply integrates instance identification into the visual geometry grounded transformer. To address the severe attention dispersion problem caused by the massive number of global image tokens, we propose the Frame-level Attention Distribution Alignment (FADA) strategy. FADA explicitly guides object queries to attend to instance-relevant frames during training, providing structured supervision without extra inference overhead. Extensive experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200, outperforming both recent joint models and RGB-D-based approaches, while exhibiting strong generalization capabilities on ScanNet++.

Related papers

• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation [76.36174247570716]
  We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models continuous and coherent geometry from pose-free multi-view images.<n> IVGT learns a continuous neural scene representation in a canonical coordinate system and supports continuous spatial queries at any 3D positions.<n>It allows direct extraction of continuous and coherent surface geometry, enabling rendering of RGB images, depth maps, and surface normal maps from arbitrary viewpoints.
  arXiv  Detail & Related papers  (2026-05-15T17:59:57Z)
• PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations [39.816194543559526]
  We propose PointForward, a feedforward driving reconstruction framework through point-aligned representations.<n>Unlike pixel-aligned methods, we aggregate multi-view image information via spatial-temporal fusion.<n>Our method enables instance-level motion propagation and temporally consistent dynamic representations.
  arXiv  Detail & Related papers  (2026-05-12T06:20:59Z)
• FAST3DIS: Feed-forward Anchored Scene Transformer for 3D Instance Segmentation [15.271467111162714]
  FAST3DIS is an end-to-end approach that effectively bypasses post-hoc clustering.<n>We introduce a 3D-anchored, query-based Transformer architecture built upon a foundational depth backbone.<n> Experiments on complex indoor 3D datasets demonstrate that our approach achieves competitive segmentation accuracy.
  arXiv  Detail & Related papers  (2026-03-27T00:45:31Z)
• MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation [59.75554954111619]
  We introduce Multi-view 3D Referring Expression (MV-3DRES), where the model must recover scene structure and segment the referred object directly from sparse multi-view images.<n>We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning.<n>Experiments show that MVGGT establishes the first strong baseline and achieves both high accuracy and fast inference, outperforming existing alternatives.
  arXiv  Detail & Related papers  (2026-01-11T11:44:07Z)
• Hierarchical Image-Guided 3D Point Cloud Segmentation in Industrial Scenes via Multi-View Bayesian Fusion [4.679314646805623]
  3D segmentation is critical for understanding complex scenes with dense layouts and multi-scale objects.<n>Existing 3D point-based methods require costly annotations, while image-guided methods often suffer from semantic inconsistencies across views.<n>We propose a hierarchical image-guided 3D segmentation framework that progressively refines segmentation from instance-level to part-level.
  arXiv  Detail & Related papers  (2025-12-07T15:15:52Z)
• IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction [82.53307702809606]
  Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions.<n>We propose InstanceGrounded Geometry Transformer (IGGT) to unify the knowledge for both spatial reconstruction and instance-level contextual understanding.
  arXiv  Detail & Related papers  (2025-10-26T14:57:44Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks [53.67497327319569]
  We introduce a novel neural rendering technique to solve image-to-3D from a single view. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.
  arXiv  Detail & Related papers  (2023-12-24T08:42:37Z)
• Anyview: Generalizable Indoor 3D Object Detection with Variable Frames [60.48134767838629]
  We present a novel 3D detection framework named AnyView for our practical applications.<n>Our method achieves both great generalizability and high detection accuracy with a simple and clean architecture.
  arXiv  Detail & Related papers  (2023-10-09T02:15:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.