Single-View View Synthesis with Self-Rectified Pseudo-Stereo

• URL: http://arxiv.org/abs/2304.09527v2
• Date: Thu, 20 Apr 2023 02:05:11 GMT
• Title: Single-View View Synthesis with Self-Rectified Pseudo-Stereo
• Authors: Yang Zhou, Hanjie Wu, Wenxi Liu, Zheng Xiong, Jing Qin, Shengfeng He
• Abstract summary: We leverage the reliable and explicit stereo prior to generate a pseudo-stereo viewpoint. We propose a self-rectified stereo synthesis to amend erroneous regions in an identify-rectify manner. Our method outperforms state-of-the-art single-view view synthesis methods and stereo synthesis methods.
• Score: 49.946151180828465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthesizing novel views from a single view image is a highly ill-posed problem. We discover an effective solution to reduce the learning ambiguity by expanding the single-view view synthesis problem to a multi-view setting. Specifically, we leverage the reliable and explicit stereo prior to generate a pseudo-stereo viewpoint, which serves as an auxiliary input to construct the 3D space. In this way, the challenging novel view synthesis process is decoupled into two simpler problems of stereo synthesis and 3D reconstruction. In order to synthesize a structurally correct and detail-preserved stereo image, we propose a self-rectified stereo synthesis to amend erroneous regions in an identify-rectify manner. Hard-to-train and incorrect warping samples are first discovered by two strategies, 1) pruning the network to reveal low-confident predictions; and 2) bidirectionally matching between stereo images to allow the discovery of improper mapping. These regions are then inpainted to form the final pseudo-stereo. With the aid of this extra input, a preferable 3D reconstruction can be easily obtained, and our method can work with arbitrary 3D representations. Extensive experiments show that our method outperforms state-of-the-art single-view view synthesis methods and stereo synthesis methods.

Related papers

• FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes [50.534213038479926]
  FreeSplat is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. We propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views.
  arXiv  Detail & Related papers  (2024-05-28T08:40:14Z)
• Generalizable Novel-View Synthesis using a Stereo Camera [21.548844864282994]
  We propose the first generalizable view synthesis approach that specifically targets multi-view stereo-camera images. We introduce stereo matching into novel-view synthesis for high-quality geometry reconstruction. Our experimental results demonstrate that StereoNeRF surpasses previous approaches in generalizable view synthesis.
  arXiv  Detail & Related papers  (2024-04-21T05:39:44Z)
• Learning to Render Novel Views from Wide-Baseline Stereo Pairs [26.528667940013598]
  We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. Existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry. We propose an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray.
  arXiv  Detail & Related papers  (2023-04-17T17:40:52Z)
• Generative Novel View Synthesis with 3D-Aware Diffusion Models [96.78397108732233]
  We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image. Our method makes use of existing 2D diffusion backbones but, crucially, incorporates geometry priors in the form of a 3D feature volume. In addition to generating novel views, our method has the ability to autoregressively synthesize 3D-consistent sequences.
  arXiv  Detail & Related papers  (2023-04-05T17:15:47Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis [72.09193030350396]
  We study the problem of view synthesis with certain amount of rotations from a pair of images, what we called stereo unstructured magnification. We propose a novel multiple homography image representation, comprising of a set of scene planes with fixed normals and distances. We derive an angle-based cost to guide the blending of multi-normal images by exploiting per-normal geometry.
  arXiv  Detail & Related papers  (2022-04-01T01:39:28Z)
• Look, Evolve and Mold: Learning 3D Shape Manifold via Single-view Synthetic Data [32.54820023526409]
  We propose a domain-adaptive network for single-view 3D reconstruction, dubbed LEM, to generalize towards the natural scenario. Experiments on several benchmarks demonstrate the effectiveness and robustness of the proposed method.
  arXiv  Detail & Related papers  (2021-03-08T14:30:18Z)
• Semantic View Synthesis [56.47999473206778]
  We tackle a new problem of semantic view synthesis -- generating free-viewpoint rendering of a synthesized scene using a semantic label map as input. First, we focus on synthesizing the color and depth of the visible surface of the 3D scene. We then use the synthesized color and depth to impose explicit constraints on the multiple-plane image (MPI) representation prediction process.
  arXiv  Detail & Related papers  (2020-08-24T17:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.