Erasing the Ephemeral: Joint Camera Refinement and Transient Object Removal for Street View Synthesis

• URL: http://arxiv.org/abs/2311.17634v1
• Date: Wed, 29 Nov 2023 13:51:12 GMT
• Title: Erasing the Ephemeral: Joint Camera Refinement and Transient Object Removal for Street View Synthesis
• Authors: Mreenav Shyam Deka and Lu Sang and Daniel Cremers
• Abstract summary: We introduce a method that tackles challenges on view synthesis for outdoor scenarios. We employ a neural point light field scene representation and strategically detect and mask out dynamic objects to reconstruct novel scenes without artifacts. We demonstrate state-of-the-art results in synthesizing novel views of urban scenes.
• Score: 44.90761677737313
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Synthesizing novel views for urban environments is crucial for tasks like autonomous driving and virtual tours. Compared to object-level or indoor situations, outdoor settings present unique challenges, such as inconsistency across frames due to moving vehicles and camera pose drift over lengthy sequences. In this paper, we introduce a method that tackles these challenges on view synthesis for outdoor scenarios. We employ a neural point light field scene representation and strategically detect and mask out dynamic objects to reconstruct novel scenes without artifacts. Moreover, we simultaneously optimize camera pose along with the view synthesis process, and thus, we simultaneously refine both elements. Through validation on real-world urban datasets, we demonstrate state-of-the-art results in synthesizing novel views of urban scenes.

Related papers

• ProSGNeRF: Progressive Dynamic Neural Scene Graph with Frequency Modulated Auto-Encoder in Urban Scenes [16.037300340326368]
  Implicit neural representation has demonstrated promising results in view synthesis for large and complex scenes. Existing approaches either fail to capture the fast-moving objects or need to build the scene graph without camera ego-motions. We aim to jointly solve the view synthesis problem of large-scale urban scenes and fast-moving vehicles.
  arXiv  Detail & Related papers  (2023-12-14T16:11:42Z)
• Fast View Synthesis of Casual Videos with Soup-of-Planes [24.35962788109883]
  Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. Our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.
  arXiv  Detail & Related papers  (2023-12-04T18:55:48Z)
• Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis [37.98068169673019]
  Implicit neural representations have shown powerful capacity in modeling real-world 3D scenes, offering superior performance in novel view synthesis. We propose a unified Neural Radiance Field (NeRF) framework to effectively perform joint scene decomposition and composition.
  arXiv  Detail & Related papers  (2023-08-05T10:42:05Z)
• LANe: Lighting-Aware Neural Fields for Compositional Scene Synthesis [65.20672798704128]
  We present Lighting-Aware Neural Field (LANe) for compositional synthesis of driving scenes. We learn a scene representation that disentangles the static background and transient elements into a world-NeRF and class-specific object-NeRFs. We demonstrate the performance of our model on a synthetic dataset of diverse lighting conditions rendered with the CARLA simulator.
  arXiv  Detail & Related papers  (2023-04-06T17:59:25Z)
• SPARF: Neural Radiance Fields from Sparse and Noisy Poses [58.528358231885846]
  We introduce Sparse Pose Adjusting Radiance Field (SPARF) to address the challenge of novel-view synthesis. Our approach exploits multi-view geometry constraints in order to jointly learn the NeRF and refine the camera poses.
  arXiv  Detail & Related papers  (2022-11-21T18:57:47Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination [63.992213016011235]
  We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
  arXiv  Detail & Related papers  (2022-07-27T16:07:48Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.