PERF: Panoramic Neural Radiance Field from a Single Panorama
- URL: http://arxiv.org/abs/2310.16831v2
- Date: Sat, 28 Oct 2023 16:50:41 GMT
- Title: PERF: Panoramic Neural Radiance Field from a Single Panorama
- Authors: Guangcong Wang and Peng Wang and Zhaoxi Chen and Wenping Wang and Chen
Change Loy and Ziwei Liu
- Abstract summary: PERF is a novel view synthesis framework that trains a panoramic neural radiance field from a single panorama.
We propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene.
Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.
- Score: 109.31072618058043
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural Radiance Field (NeRF) has achieved substantial progress in novel view
synthesis given multi-view images. Recently, some works have attempted to train
a NeRF from a single image with 3D priors. They mainly focus on a limited field
of view with a few occlusions, which greatly limits their scalability to
real-world 360-degree panoramic scenarios with large-size occlusions. In this
paper, we present PERF, a 360-degree novel view synthesis framework that trains
a panoramic neural radiance field from a single panorama. Notably, PERF allows
3D roaming in a complex scene without expensive and tedious image collection.
To achieve this goal, we propose a novel collaborative RGBD inpainting method
and a progressive inpainting-and-erasing method to lift up a 360-degree 2D
scene to a 3D scene. Specifically, we first predict a panoramic depth map as
initialization given a single panorama and reconstruct visible 3D regions with
volume rendering. Then we introduce a collaborative RGBD inpainting approach
into a NeRF for completing RGB images and depth maps from random views, which
is derived from an RGB Stable Diffusion model and a monocular depth estimator.
Finally, we introduce an inpainting-and-erasing strategy to avoid inconsistent
geometry between a newly-sampled view and reference views. The two components
are integrated into the learning of NeRFs in a unified optimization framework
and achieve promising results. Extensive experiments on Replica and a new
dataset PERF-in-the-wild demonstrate the superiority of our PERF over
state-of-the-art methods. Our PERF can be widely used for real-world
applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization
applications. Project page and code are available at
https://perf-project.github.io/ and https://github.com/perf-project/PeRF.
Related papers
- Pano2Room: Novel View Synthesis from a Single Indoor Panorama [20.262621556667852]
Pano2Room is designed to automatically reconstruct high-quality 3D indoor scenes from a single panoramic image.
The key idea is to initially construct a preliminary mesh from the input panorama, and iteratively refine this mesh using a panoramic RGBD inpainter.
The refined mesh is converted into a 3D Gaussian Splatting field and trained with the collected pseudo novel views.
arXiv Detail & Related papers (2024-08-21T08:19:12Z) - DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes.
Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
arXiv Detail & Related papers (2024-06-17T21:15:13Z) - DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting [56.101576795566324]
We present a text-to-3D 360$circ$ scene generation pipeline.
Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement.
Our method offers a globally consistent 3D scene within a 360$circ$ perspective.
arXiv Detail & Related papers (2024-04-10T10:46:59Z) - Denoising Diffusion via Image-Based Rendering [54.20828696348574]
We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes.
First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes.
Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
arXiv Detail & Related papers (2024-02-05T19:00:45Z) - Reference-guided Controllable Inpainting of Neural Radiance Fields [26.296017756560467]
We focus on inpainting regions in a view-consistent and controllable manner.
We use monocular depth estimators to back-project the inpainted view to the correct 3D positions.
For non-reference disoccluded regions, we devise a method based on image inpainters to guide both the geometry and appearance.
arXiv Detail & Related papers (2023-04-19T14:11:21Z) - SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance
Fields [19.740018132105757]
SceneRF is a self-supervised monocular scene reconstruction method using only posed image sequences for training.
At inference, a single input image suffices to hallucinate novel depth views which are fused together to obtain 3D scene reconstruction.
arXiv Detail & Related papers (2022-12-05T18:59:57Z) - Vision Transformer for NeRF-Based View Synthesis from a Single Input
Image [49.956005709863355]
We propose to leverage both the global and local features to form an expressive 3D representation.
To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering.
Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
arXiv Detail & Related papers (2022-07-12T17:52:04Z) - Deep Multi Depth Panoramas for View Synthesis [70.9125433400375]
We present a novel scene representation - Multi Depth Panorama (MDP) - that consists of multiple RGBD$alpha$ panoramas.
MDPs are more compact than previous 3D scene representations and enable high-quality, efficient new view rendering.
arXiv Detail & Related papers (2020-08-04T20:29:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.