Tiled Multiplane Images for Practical 3D Photography
- URL: http://arxiv.org/abs/2309.14291v1
- Date: Mon, 25 Sep 2023 16:56:40 GMT
- Title: Tiled Multiplane Images for Practical 3D Photography
- Authors: Numair Khan, Douglas Lanman, Lei Xiao
- Abstract summary: A Multiplane Image (MPI) estimates the scene as a stack of RGBA layers.
Unlike neural radiance fields, an MPI can be efficiently rendered on graphics hardware.
We propose a method for generating a TMPI with adaptive depth planes for single-view 3D photography in the wild.
- Score: 9.309697339467148
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The task of synthesizing novel views from a single image has useful
applications in virtual reality and mobile computing, and a number of
approaches to the problem have been proposed in recent years. A Multiplane
Image (MPI) estimates the scene as a stack of RGBA layers, and can model
complex appearance effects, anti-alias depth errors and synthesize soft edges
better than methods that use textured meshes or layered depth images. And
unlike neural radiance fields, an MPI can be efficiently rendered on graphics
hardware. However, MPIs are highly redundant and require a large number of
depth layers to achieve plausible results. Based on the observation that the
depth complexity in local image regions is lower than that over the entire
image, we split an MPI into many small, tiled regions, each with only a few
depth planes. We call this representation a Tiled Multiplane Image (TMPI). We
propose a method for generating a TMPI with adaptive depth planes for
single-view 3D photography in the wild. Our synthesized results are comparable
to state-of-the-art single-view MPI methods while having lower computational
overhead.
Related papers
- CUBE360: Learning Cubic Field Representation for Monocular 360 Depth Estimation for Virtual Reality [32.023283261191104]
CUBE360 learns a cubic field composed of multiple MPIs from a single panoramic image for depth estimation at any view direction.
Experiments on both synthetic and real-world datasets demonstrate the superior performance of CUBE360 compared to prior SSL methods.
arXiv Detail & Related papers (2024-10-08T06:52:46Z) - MMPI: a Flexible Radiance Field Representation by Multiple Multi-plane
Images Blending [61.45757368117578]
This paper presents a flexible representation of neural radiance fields based on multi-plane images (MPI)
MPI is widely used in NeRF learning for its simple definition, easy calculation, and powerful ability to represent unbounded scenes.
We show that MPI can synthesize high-quality novel views of complex scenes with diverse camera distributions and view directions.
arXiv Detail & Related papers (2023-09-30T04:36:43Z) - Structural Multiplane Image: Bridging Neural View Synthesis and 3D
Reconstruction [39.89856628467095]
We introduce the Structural MPI (S-MPI), where the plane structure approximates 3D scenes concisely.
Despite the intuition and demand of applying S-MPI, great challenges are introduced, e.g., high-fidelity approximation for both RGBA layers and plane poses.
Our method outperforms both previous state-of-the-art MPI-based view synthesis methods and planar reconstruction methods.
arXiv Detail & Related papers (2023-03-10T14:18:40Z) - High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views.
Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
arXiv Detail & Related papers (2022-11-28T18:59:52Z) - Single-View View Synthesis in the Wild with Learned Adaptive Multiplane
Images [15.614631883233898]
Existing methods have shown promising results leveraging monocular depth estimation and color inpainting with layered depth representations.
We propose a new method based on the multiplane image (MPI) representation.
The experiments on both synthetic and real datasets demonstrate that our trained model works remarkably well and achieves state-of-the-art results.
arXiv Detail & Related papers (2022-05-24T02:57:16Z) - Remote Sensing Novel View Synthesis with Implicit Multiplane
Representations [26.33490094119609]
We propose a novel remote sensing view synthesis method by leveraging the recent advances in implicit neural representations.
Considering the overhead and far depth imaging of remote sensing images, we represent the 3D space by combining implicit multiplane images (MPI) representation and deep neural networks.
Images from any novel views can be freely rendered on the basis of the reconstructed model.
arXiv Detail & Related papers (2022-05-18T13:03:55Z) - Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo [103.08512487830669]
We present a modern solution to the multi-view photometric stereo problem (MVPS)
We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry.
Our method performs neural rendering of multi-view images while utilizing surface normals estimated by a deep photometric stereo network.
arXiv Detail & Related papers (2021-10-11T20:20:03Z) - Deep Multi Depth Panoramas for View Synthesis [70.9125433400375]
We present a novel scene representation - Multi Depth Panorama (MDP) - that consists of multiple RGBD$alpha$ panoramas.
MDPs are more compact than previous 3D scene representations and enable high-quality, efficient new view rendering.
arXiv Detail & Related papers (2020-08-04T20:29:15Z) - 3D Photography using Context-aware Layered Depth Inpainting [50.66235795163143]
We propose a method for converting a single RGB-D input image into a 3D photo.
A learning-based inpainting model synthesizes new local color-and-depth content into the occluded region.
The resulting 3D photos can be efficiently rendered with motion parallax.
arXiv Detail & Related papers (2020-04-09T17:59:06Z) - Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object.
We first estimate per-view depth maps using a deep multi-view stereo network.
These depth maps are used to coarsely align the different views.
We propose a novel multi-view reflectance estimation network architecture.
arXiv Detail & Related papers (2020-03-27T21:28:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.