ProSGNeRF: Progressive Dynamic Neural Scene Graph with Frequency Modulated Auto-Encoder in Urban Scenes

• URL: http://arxiv.org/abs/2312.09076v2
• Date: Fri, 15 Dec 2023 07:11:28 GMT
• Title: ProSGNeRF: Progressive Dynamic Neural Scene Graph with Frequency Modulated Auto-Encoder in Urban Scenes
• Authors: Tianchen Deng, Siyang Liu, Xuan Wang, Yejia Liu, Danwei Wang, Weidong Chen
• Abstract summary: Implicit neural representation has demonstrated promising results in view synthesis for large and complex scenes. Existing approaches either fail to capture the fast-moving objects or need to build the scene graph without camera ego-motions. We aim to jointly solve the view synthesis problem of large-scale urban scenes and fast-moving vehicles.
• Score: 16.037300340326368
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Implicit neural representation has demonstrated promising results in view synthesis for large and complex scenes. However, existing approaches either fail to capture the fast-moving objects or need to build the scene graph without camera ego-motions, leading to low-quality synthesized views of the scene. We aim to jointly solve the view synthesis problem of large-scale urban scenes and fast-moving vehicles, which is more practical and challenging. To this end, we first leverage a graph structure to learn the local scene representations of dynamic objects and the background. Then, we design a progressive scheme that dynamically allocates a new local scene graph trained with frames within a temporal window, allowing us to scale up the representation to an arbitrarily large scene. Besides, the training views of urban scenes are relatively sparse, which leads to a significant decline in reconstruction accuracy for dynamic objects. Therefore, we design a frequency auto-encoder network to encode the latent code and regularize the frequency range of objects, which can enhance the representation of dynamic objects and address the issue of sparse image inputs. Additionally, we employ lidar point projection to maintain geometry consistency in large-scale urban scenes. Experimental results demonstrate that our method achieves state-of-the-art view synthesis accuracy, object manipulation, and scene roaming ability. The code will be open-sourced upon paper acceptance.

Related papers

• DENSER: 3D Gaussians Splatting for Scene Reconstruction of Dynamic Urban Environments [0.0]
  We propose DENSER, a framework that significantly enhances the representation of dynamic objects. The proposed approach significantly outperforms state-of-the-art methods by a wide margin.
  arXiv  Detail & Related papers  (2024-09-16T07:11:58Z)
• D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
  This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures. Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
  arXiv  Detail & Related papers  (2024-06-14T14:35:44Z)
• Multi-Level Neural Scene Graphs for Dynamic Urban Environments [64.26401304233843]
  We present a novel, decomposable radiance field approach for dynamic urban environments. We propose a multi-level neural scene graph representation that scales to thousands of images from dozens of sequences with hundreds of fast-moving objects.
  arXiv  Detail & Related papers  (2024-03-29T21:52:01Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• Editable Free-viewpoint Video Using a Layered Neural Representation [35.44420164057911]
  We propose the first approach for editable free-viewpoint video generation for large-scale dynamic scenes using only sparse 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity including the environment itself is formulated into a space-time coherent neural layered radiance representation called ST-NeRF. Experiments demonstrate the effectiveness of our approach to achieve high-quality, photo-realistic, and editable free-viewpoint video generation for dynamic scenes.
  arXiv  Detail & Related papers  (2021-04-30T06:50:45Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes [70.76742458931935]
  We introduce a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for complex dynamic scenes, including thin structures, view-dependent effects, and natural degrees of motion.
  arXiv  Detail & Related papers  (2020-11-26T01:23:44Z)
• Neural Scene Graphs for Dynamic Scenes [57.65413768984925]
  We present the first neural rendering method that decomposes dynamic scenes into scene graphs. We learn implicitly encoded scenes, combined with a jointly learned latent representation to describe objects with a single implicit function.
  arXiv  Detail & Related papers  (2020-11-20T12:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.