SAMPLING: Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image

• URL: http://arxiv.org/abs/2309.06323v2
• Date: Wed, 13 Sep 2023 05:43:53 GMT
• Title: SAMPLING: Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image
• Authors: Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, Ming-Hsuan Yang
• Abstract summary: We introduce SAMPLING, a Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image. Our method demonstrates considerable performance gains in large-scale unbounded outdoor scenes using a single image on the KITTI dataset.
• Score: 60.52991173059486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent novel view synthesis methods obtain promising results for relatively small scenes, e.g., indoor environments and scenes with a few objects, but tend to fail for unbounded outdoor scenes with a single image as input. In this paper, we introduce SAMPLING, a Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image based on improved multiplane images (MPI). Observing that depth distribution varies significantly for unbounded outdoor scenes, we employ an adaptive-bins strategy for MPI to arrange planes in accordance with each scene image. To represent intricate geometry and multi-scale details, we further introduce a hierarchical refinement branch, which results in high-quality synthesized novel views. Our method demonstrates considerable performance gains in synthesizing large-scale unbounded outdoor scenes using a single image on the KITTI dataset and generalizes well to the unseen Tanks and Temples dataset.The code and models will soon be made available.

Related papers

• MultiDiff: Consistent Novel View Synthesis from a Single Image [60.04215655745264]
  MultiDiff is a novel approach for consistent novel view synthesis of scenes from a single RGB image. Our results demonstrate that MultiDiff outperforms state-of-the-art methods on the challenging, real-world datasets RealEstate10K and ScanNet.
  arXiv  Detail & Related papers  (2024-06-26T17:53:51Z)
• MMPI: a Flexible Radiance Field Representation by Multiple Multi-plane Images Blending [61.45757368117578]
  This paper presents a flexible representation of neural radiance fields based on multi-plane images (MPI) MPI is widely used in NeRF learning for its simple definition, easy calculation, and powerful ability to represent unbounded scenes. We show that MPI can synthesize high-quality novel views of complex scenes with diverse camera distributions and view directions.
  arXiv  Detail & Related papers  (2023-09-30T04:36:43Z)
• Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images [15.614631883233898]
  Existing methods have shown promising results leveraging monocular depth estimation and color inpainting with layered depth representations. We propose a new method based on the multiplane image (MPI) representation. The experiments on both synthetic and real datasets demonstrate that our trained model works remarkably well and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2022-05-24T02:57:16Z)
• Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations [48.05445941939446]
  A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. We propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets.
  arXiv  Detail & Related papers  (2021-11-25T16:18:56Z)
• Free View Synthesis [100.86844680362196]
  We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.
  arXiv  Detail & Related papers  (2020-08-12T18:16:08Z)
• Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision [83.33283892171562]
  Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically different from the given label map. We propose an end-to-end network for joint global and local feature alignment and synthesis.
  arXiv  Detail & Related papers  (2020-04-18T18:17:40Z)
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis [78.5281048849446]
  We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses.
  arXiv  Detail & Related papers  (2020-03-19T17:57:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.