FlowIBR: Leveraging Pre-Training for Efficient Neural Image-Based Rendering of Dynamic Scenes

• URL: http://arxiv.org/abs/2309.05418v2
• Date: Mon, 15 Apr 2024 13:42:13 GMT
• Title: FlowIBR: Leveraging Pre-Training for Efficient Neural Image-Based Rendering of Dynamic Scenes
• Authors: Marcel Büsching, Josef Bengtson, David Nilsson, Mårten Björkman,
• Abstract summary: FlowIBR is a novel approach for efficient monocular novel view of dynamic scenes. It integrates a neural image-based rendering method, pre-trained on a large corpus of widely available static scenes, with a per-scene optimized scene flow field. The proposed method reduces per-scene optimization time by an order of magnitude, achieving comparable rendering quality to existing methods.
• Score: 5.118560450410779
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce FlowIBR, a novel approach for efficient monocular novel view synthesis of dynamic scenes. Existing techniques already show impressive rendering quality but tend to focus on optimization within a single scene without leveraging prior knowledge, resulting in long optimization times per scene. FlowIBR circumvents this limitation by integrating a neural image-based rendering method, pre-trained on a large corpus of widely available static scenes, with a per-scene optimized scene flow field. Utilizing this flow field, we bend the camera rays to counteract the scene dynamics, thereby presenting the dynamic scene as if it were static to the rendering network. The proposed method reduces per-scene optimization time by an order of magnitude, achieving comparable rendering quality to existing methods -- all on a single consumer-grade GPU.

Related papers

• D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
  This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures. Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
  arXiv  Detail & Related papers  (2024-06-14T14:35:44Z)
• Stylizing Sparse-View 3D Scenes with Hierarchical Neural Representation [0.0]
  A surge of 3D style transfer methods has been proposed that leverage the scene reconstruction power of a pre-trained neural radiance field (NeRF) In this paper, we consider the stylization of sparse-view scenes in terms of disentangling content semantics and style textures. A novel hierarchical encoding-based neural representation is designed to generate high-quality stylized scenes directly from implicit scene representations.
  arXiv  Detail & Related papers  (2024-04-08T07:01:42Z)
• Multi-Level Neural Scene Graphs for Dynamic Urban Environments [64.26401304233843]
  We present a novel, decomposable radiance field approach for dynamic urban environments. We propose a multi-level neural scene graph representation that scales to thousands of images from dozens of sequences with hundreds of fast-moving objects.
  arXiv  Detail & Related papers  (2024-03-29T21:52:01Z)
• VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
  We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
  arXiv  Detail & Related papers  (2024-02-27T11:40:50Z)
• Deep Dynamic Scene Deblurring from Optical Flow [53.625999196063574]
  Deblurring can provide visually more pleasant pictures and make photography more convenient. It is difficult to model the non-uniform blur mathematically. We develop a convolutional neural network (CNN) to restore the sharp images from the deblurred features.
  arXiv  Detail & Related papers  (2023-01-18T06:37:21Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• NPBG++: Accelerating Neural Point-Based Graphics [14.366073496519139]
  NPBG++ is a novel view synthesis (NVS) task that achieves high rendering realism with low scene fitting time. Our method efficiently leverages the multiview observations and the point cloud of a static scene to predict a neural descriptor for each point. In our comparisons, the proposed system outperforms previous NVS approaches in terms of fitting and rendering runtimes while producing images of similar quality.
  arXiv  Detail & Related papers  (2022-03-24T19:59:39Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.