Self-improving Multiplane-to-layer Images for Novel View Synthesis

• URL: http://arxiv.org/abs/2210.01602v1
• Date: Tue, 4 Oct 2022 13:27:14 GMT
• Title: Self-improving Multiplane-to-layer Images for Novel View Synthesis
• Authors: Pavel Solovev, Taras Khakhulin, Denis Korzhenkov
• Abstract summary: We present a new method for lightweight novel-view synthesis that generalizes to an arbitrary forward-facing scene. We start by representing the scene with a set of fronto-parallel semitransparent planes and afterward convert them to deformable layers in an end-to-end manner. Our method does not require fine-tuning when a new scene is processed and can handle an arbitrary number of views without restrictions.
• Score: 3.9901365062418312
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present a new method for lightweight novel-view synthesis that generalizes to an arbitrary forward-facing scene. Recent approaches are computationally expensive, require per-scene optimization, or produce a memory-expensive representation. We start by representing the scene with a set of fronto-parallel semitransparent planes and afterward convert them to deformable layers in an end-to-end manner. Additionally, we employ a feed-forward refinement procedure that corrects the estimated representation by aggregating information from input views. Our method does not require fine-tuning when a new scene is processed and can handle an arbitrary number of views without restrictions. Experimental results show that our approach surpasses recent models in terms of common metrics and human evaluation, with the noticeable advantage in inference speed and compactness of the inferred layered geometry, see https://samsunglabs.github.io/MLI

Related papers

• Efficient Depth-Guided Urban View Synthesis [52.841803876653465]
  We introduce Efficient Depth-Guided Urban View Synthesis (EDUS) for fast feed-forward inference and efficient per-scene fine-tuning. EDUS exploits noisy predicted geometric priors as guidance to enable generalizable urban view synthesis from sparse input images. Our results indicate that EDUS achieves state-of-the-art performance in sparse view settings when combined with fast test-time optimization.
  arXiv  Detail & Related papers  (2024-07-17T08:16:25Z)
• UpFusion: Novel View Diffusion from Unposed Sparse View Observations [66.36092764694502]
  UpFusion can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images. We show that this mechanism allows generating high-fidelity novel views while improving the synthesis quality given additional (unposed) images.
  arXiv  Detail & Related papers  (2023-12-11T18:59:55Z)
• SAMPLING: Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image [60.52991173059486]
  We introduce SAMPLING, a Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image. Our method demonstrates considerable performance gains in large-scale unbounded outdoor scenes using a single image on the KITTI dataset.
  arXiv  Detail & Related papers  (2023-09-12T15:33:09Z)
• Fast Non-Rigid Radiance Fields from Monocularized Data [66.74229489512683]
  This paper proposes a new method for full 360deg inward-facing novel view synthesis of non-rigidly deforming scenes. At the core of our method are 1) An efficient deformation module that decouples the processing of spatial and temporal information for accelerated training and inference; and 2) A static module representing the canonical scene as a fast hash-encoded neural radiance field. In both cases, our method is significantly faster than previous methods, converging in less than 7 minutes and achieving real-time framerates at 1K resolution, while obtaining a higher visual accuracy for generated novel views.
  arXiv  Detail & Related papers  (2022-12-02T18:51:10Z)
• Generalizable Patch-Based Neural Rendering [46.41746536545268]
  We propose a new paradigm for learning models that can synthesize novel views of unseen scenes. Our method is capable of predicting the color of a target ray in a novel scene directly, just from a collection of patches sampled from the scene. We show that our approach outperforms the state-of-the-art on novel view synthesis of unseen scenes even when being trained with considerably less data than prior work.
  arXiv  Detail & Related papers  (2022-07-21T17:57:04Z)
• Stereo Magnification with Multi-Layer Images [7.20447989181373]
  We introduce a new view synthesis approach based on multiple semi-transparent layers with scene-adapted geometry. Our approach infers such representations from stereo pairs in two stages. In experiments, we demonstrate the advantage of the proposed approach over the use of regularly-spaced layers.
  arXiv  Detail & Related papers  (2022-01-13T15:19:46Z)
• Free View Synthesis [100.86844680362196]
  We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.
  arXiv  Detail & Related papers  (2020-08-12T18:16:08Z)
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis [78.5281048849446]
  We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses.
  arXiv  Detail & Related papers  (2020-03-19T17:57:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.