Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

• URL: http://arxiv.org/abs/2607.19765v1
• Date: Wed, 22 Jul 2026 05:16:30 GMT
• Title: Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation
• Abstract summary: We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views.<n>On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB.
• Score: 30.949009648845134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance. When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure. These results indicate that the correspondence learned for RGB view synthesis can also propagate view-independent per-pixel labels. From this observation, we present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views, without 3D reconstruction or any segmentation-specific training of the view synthesis model. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation. On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB. The label propagation also transfers across datasets, surpassing these approaches on Replica without any fine-tuning.

Related papers

• SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis [42.23526579547921]
  SplatGuide reuses a single 3DGS scene across three complementary roles.<n>Per-Gaussian source-view indices are rendered into a target-view voting map.<n>Reconstruction tokens supply feature-level guidance via cross-attention.
  arXiv  Detail & Related papers  (2026-08-17T17:45:57Z)
• AR-1-to-3: Single Image to Consistent 3D Object Generation via Next-View Prediction [69.65671384868344]
  We propose AR-1-to-3, a novel next-view prediction paradigm based on diffusion models.<n>We show that our method significantly improves the consistency between the generated views and the input views, producing high-fidelity 3D assets.
  arXiv  Detail & Related papers  (2025-03-17T08:39:10Z)
• LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias [50.13457154615262]
  We propose a transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs.<n>We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs.
  arXiv  Detail & Related papers  (2024-10-22T17:58:28Z)
• FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes [50.534213038479926]
  FreeSplat is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. We propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views.
  arXiv  Detail & Related papers  (2024-05-28T08:40:14Z)
• GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding [30.951440204237166]
  We introduce a Generalizable Semantic Neural Radiance Field (GSNeRF), which takes image semantics into the synthesis process. Our GSNeRF is composed of two stages: Semantic Geo-Reasoning and Depth-Guided Visual rendering.
  arXiv  Detail & Related papers  (2024-03-06T10:55:50Z)
• UpFusion: Novel View Diffusion from Unposed Sparse View Observations [66.36092764694502]
  UpFusion can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images. We show that this mechanism allows generating high-fidelity novel views while improving the synthesis quality given additional (unposed) images.
  arXiv  Detail & Related papers  (2023-12-11T18:59:55Z)
• Learning to Render Novel Views from Wide-Baseline Stereo Pairs [26.528667940013598]
  We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. Existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry. We propose an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray.
  arXiv  Detail & Related papers  (2023-04-17T17:40:52Z)
• Street-view Panoramic Video Synthesis from a Single Satellite Image [92.26826861266784]
  We present a novel method for synthesizing both temporally and geometrically consistent street-view panoramic video. Existing cross-view synthesis approaches focus more on images, while video synthesis in such a case has not yet received enough attention.
  arXiv  Detail & Related papers  (2020-12-11T20:22:38Z)
• Stable View Synthesis [100.86844680362196]
  We present Stable View Synthesis (SVS) Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene. SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets.
  arXiv  Detail & Related papers  (2020-11-14T07:24:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.