Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding

• URL: http://arxiv.org/abs/2608.29177v1
• Date: Sat, 29 Aug 2026 09:58:24 GMT
• Title: Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding
• Abstract summary: SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
• Score: 63.599342750566485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of novel view synthesis (NVS) and open-vocabulary segmentation (OVS) has recently yielded powerful feed-forward 3D foundation models. However, their inherent reliance on static-scene assumptions leads to severe misalignment of spatial features in unconstrained dynamic environments. To bridge this critical gap, we propose SPAR, a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation. Furthermore, we introduce a dynamic-region-aware end-to-end training paradigm that structurally couples motion estimation with multi-view visual and semantic learning. This unified approach enables the network to inherently resolve motion conflicts and distill multi-view consistent, temporally stable scene representations from dynamic inputs. Extensive experiments on the challenging D-RE10K benchmark demonstrate that SPAR achieves state-of-the-art performance. Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively. Despite being trained in a self-supervised manner, our model achieves an mIoU of 88.5% for motion mask prediction. Furthermore, our analysis reveals a strong inter-task synergy between photometric scene reconstruction and semantic understanding, where semantic synthesis learning consistently enhances photometric fidelity in novel view rendering. Code will be available at https://github.com/dmucby/SPAR.

Related papers

• SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video [88.79625979053873]
  We propose a novel framework, SpaceEra, for pre-trained vision-language models (VLMs)<n>We extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference.<n>In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations.
  arXiv  Detail & Related papers  (2026-07-02T06:56:29Z)
• Beyond Static Gaussians: An Empirical Investigation of Architectural Paradigms for Dynamic 3D Scene Reconstruction [11.523616541218878]
  Dynamic scene reconstruction via 3D Gaussian Splatting (3DGS) has emerged as a compelling approach for representing evolving environments.<n>This paper presents a comprehensive analysis of dynamic 3DGS methods, categorizing them into two paradigms.
  arXiv  Detail & Related papers  (2026-05-30T00:42:38Z)
• SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes [9.44079073767086]
  SLARM is a feed-forward model that unifies dynamic scene reconstruction, semantic understanding, and real-time streaming inference.<n>It captures complex, non-uniform motion through higher-order motion modeling, trained solely on differentiable renderings without flow supervision.<n> SLARM processes image sequences using window-based causal attention, achieving stable, low-latency streaming inference without accumulating memory cost.
  arXiv  Detail & Related papers  (2026-03-24T07:42:05Z)
• SemanticNVS: Improving Semantic Scene Understanding in Generative Novel View Synthesis [25.524477911101325]
  We present SemanticNVS, a camera-conditioned multi-view diffusion model for novel view synthesis (NVS)<n>Existing NVS methods generate semantically implausible and distorted images under long-range camera motion.<n>We propose to integrate pre-trained semantic feature extractors to incorporate stronger scene semantics as conditioning to achieve high-quality generation even at distant viewpoints.
  arXiv  Detail & Related papers  (2026-02-23T17:45:21Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• HBSplat: Robust Sparse-View Gaussian Reconstruction with Hybrid-Loss Guided Depth and Bidirectional Warping [11.035994094874141]
  HBSplat is a framework that seamlessly integrates robust structural cues, virtual view constraints, and occluded region completion.<n> HBSplat sets a new state-of-the-art, achieving up to 21.13 dB PSNR and 0.189 LPIPS, while maintaining real-time inference.
  arXiv  Detail & Related papers  (2025-09-29T15:03:31Z)
• Dynamic View Synthesis as an Inverse Problem [3.7599363231894185]
  We address dynamic view synthesis from monocular videos as an inverse problem in a training-free setting.<n>We introduce a novel noise representation, termed K-order Recursive Noise Representation.<n>To synthesize newly visible regions from camera motion, we introduce Latent Modulation.
  arXiv  Detail & Related papers  (2025-06-09T17:59:47Z)
• SHaDe: Compact and Consistent Dynamic 3D Reconstruction via Tri-Plane Deformation and Latent Diffusion [0.0]
  We present a novel framework for dynamic 3D scene reconstruction that integrates three key components.<n>An explicit tri-plane deformation field, a view-conditioned canonical field with spherical harmonics (SH) attention, and a temporally-aware latent diffusion prior.<n>Our method encodes 4D scenes using three 2D feature planes that evolve over time, enabling efficient compact representation.
  arXiv  Detail & Related papers  (2025-05-22T11:25:38Z)
• EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splatting [108.15136508964011]
  EgoSplat is a language-embedded 3D Gaussian Splatting framework for open-vocabulary egocentric scene understanding.<n>EgoSplat achieves state-of-the-art performance in both localization and segmentation tasks on two datasets.
  arXiv  Detail & Related papers  (2025-03-14T12:21:26Z)
• S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR [50.435592120607815]
  Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR) Previous works have primarily relied on multi-stage learning, where the generated semantic scene graphs depend on intermediate processes with pose estimation and object detection. In this study, we introduce a novel single-stage bi-modal transformer framework for SGG in the OR, termed S2Former-OR.
  arXiv  Detail & Related papers  (2024-02-22T11:40:49Z)
• Scene Synthesis via Uncertainty-Driven Attribute Synchronization [52.31834816911887]
  This paper introduces a novel neural scene synthesis approach that can capture diverse feature patterns of 3D scenes. Our method combines the strength of both neural network-based and conventional scene synthesis approaches.
  arXiv  Detail & Related papers  (2021-08-30T19:45:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.