Generalizable Novel-View Synthesis using a Stereo Camera

• URL: http://arxiv.org/abs/2404.13541v1
• Date: Sun, 21 Apr 2024 05:39:44 GMT
• Title: Generalizable Novel-View Synthesis using a Stereo Camera
• Authors: Haechan Lee, Wonjoon Jin, Seung-Hwan Baek, Sunghyun Cho,
• Abstract summary: We propose the first generalizable view synthesis approach that specifically targets multi-view stereo-camera images. We introduce stereo matching into novel-view synthesis for high-quality geometry reconstruction. Our experimental results demonstrate that StereoNeRF surpasses previous approaches in generalizable view synthesis.
• Score: 21.548844864282994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose the first generalizable view synthesis approach that specifically targets multi-view stereo-camera images. Since recent stereo matching has demonstrated accurate geometry prediction, we introduce stereo matching into novel-view synthesis for high-quality geometry reconstruction. To this end, this paper proposes a novel framework, dubbed StereoNeRF, which integrates stereo matching into a NeRF-based generalizable view synthesis approach. StereoNeRF is equipped with three key components to effectively exploit stereo matching in novel-view synthesis: a stereo feature extractor, a depth-guided plane-sweeping, and a stereo depth loss. Moreover, we propose the StereoNVS dataset, the first multi-view dataset of stereo-camera images, encompassing a wide variety of both real and synthetic scenes. Our experimental results demonstrate that StereoNeRF surpasses previous approaches in generalizable view synthesis.

Related papers

• Revisiting Depth Completion from a Stereo Matching Perspective for Cross-domain Generalization [40.25292494550211]
  This paper exploits the generalization capability of modern stereo networks to face depth completion. Any stereo network or traditional stereo matcher can be seamlessly plugged into our framework.
  arXiv  Detail & Related papers  (2023-12-14T18:59:58Z)
• Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs [57.492124844326206]
  This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks.
  arXiv  Detail & Related papers  (2023-12-12T13:22:44Z)
• Mono-to-stereo through parametric stereo generation [21.502860265488216]
  We propose to convert mono to stereo by means of predicting parametric stereo parameters. In combination with PS, we also propose to model the task with generative approaches. We provide evidence that the proposed PS-based models outperform a competitive classical decorrelation baseline.
  arXiv  Detail & Related papers  (2023-06-26T12:33:29Z)
• Single-View View Synthesis with Self-Rectified Pseudo-Stereo [49.946151180828465]
  We leverage the reliable and explicit stereo prior to generate a pseudo-stereo viewpoint. We propose a self-rectified stereo synthesis to amend erroneous regions in an identify-rectify manner. Our method outperforms state-of-the-art single-view view synthesis methods and stereo synthesis methods.
  arXiv  Detail & Related papers  (2023-04-19T09:36:13Z)
• Learning to Render Novel Views from Wide-Baseline Stereo Pairs [26.528667940013598]
  We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. Existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry. We propose an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray.
  arXiv  Detail & Related papers  (2023-04-17T17:40:52Z)
• Novel-View Acoustic Synthesis [140.1107768313269]
  We introduce the novel-view acoustic synthesis (NVAS) task. given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint? We propose a neural rendering approach: Visually-Guided Acoustic Synthesis (ViGAS) network that learns to synthesize the sound of an arbitrary point in space.
  arXiv  Detail & Related papers  (2023-01-20T18:49:58Z)
• Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis [72.09193030350396]
  We study the problem of view synthesis with certain amount of rotations from a pair of images, what we called stereo unstructured magnification. We propose a novel multiple homography image representation, comprising of a set of scene planes with fixed normals and distances. We derive an angle-based cost to guide the blending of multi-normal images by exploiting per-normal geometry.
  arXiv  Detail & Related papers  (2022-04-01T01:39:28Z)
• Street-view Panoramic Video Synthesis from a Single Satellite Image [92.26826861266784]
  We present a novel method for synthesizing both temporally and geometrically consistent street-view panoramic video. Existing cross-view synthesis approaches focus more on images, while video synthesis in such a case has not yet received enough attention.
  arXiv  Detail & Related papers  (2020-12-11T20:22:38Z)
• Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo [52.68109922159688]
  We introduce a novel differentiable image formation model for active stereo, relying on both wave and geometric optics, and a novel trinocular reconstruction network. The jointly optimized pattern, which we dub "Polka Lines," together with the reconstruction network, achieve state-of-the-art active-stereo depth estimates across imaging conditions.
  arXiv  Detail & Related papers  (2020-11-26T04:02:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.