Learning a Diffusion Prior for NeRFs

• URL: http://arxiv.org/abs/2304.14473v1
• Date: Thu, 27 Apr 2023 19:24:21 GMT
• Title: Learning a Diffusion Prior for NeRFs
• Authors: Guandao Yang, Abhijit Kundu, Leonidas J. Guibas, Jonathan T. Barron, Ben Poole
• Abstract summary: We propose to use a diffusion model to generate NeRFs encoded on a regularized grid. We show that our model can sample realistic NeRFs, while at the same time allowing conditional generations, given a certain observation as guidance.
• Score: 84.99454404653339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Radiance Fields (NeRFs) have emerged as a powerful neural 3D representation for objects and scenes derived from 2D data. Generating NeRFs, however, remains difficult in many scenarios. For instance, training a NeRF with only a small number of views as supervision remains challenging since it is an under-constrained problem. In such settings, it calls for some inductive prior to filter out bad local minima. One way to introduce such inductive priors is to learn a generative model for NeRFs modeling a certain class of scenes. In this paper, we propose to use a diffusion model to generate NeRFs encoded on a regularized grid. We show that our model can sample realistic NeRFs, while at the same time allowing conditional generations, given a certain observation as guidance.

Related papers

• Self-Evolving Neural Radiance Fields [31.124406548504794]
  We propose a novel framework, dubbed Self-Evolving Neural Radiance Fields (SE-NeRF), that applies a self-training framework to neural radiance field (NeRF) We formulate few-shot NeRF into a teacher-student framework to guide the network to learn a more robust representation of the scene. We show and evaluate that applying our self-training framework to existing models improves the quality of the rendered images and achieves state-of-the-art performance in multiple settings.
  arXiv  Detail & Related papers  (2023-12-02T02:28:07Z)
• SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions [6.9980855647933655]
  supervising the depth estimated by the NeRF helps train it effectively with fewer views. We design augmented models that encourage simpler solutions by exploring the role of positional encoding and view-dependent radiance. We achieve state-of-the-art view-synthesis performance on two popular datasets by employing the above regularizations.
  arXiv  Detail & Related papers  (2023-09-07T18:02:57Z)
• DReg-NeRF: Deep Registration for Neural Radiance Fields [66.69049158826677]
  We propose DReg-NeRF to solve the NeRF registration problem on object-centric annotated scenes without human intervention. Our proposed method beats the SOTA point cloud registration methods by a large margin.
  arXiv  Detail & Related papers  (2023-08-18T08:37:49Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• Instance Neural Radiance Field [62.152611795824185]
  This paper presents one of the first learning-based NeRF 3D instance segmentation pipelines, dubbed as Instance Neural Radiance Field. We adopt a 3D proposal-based mask prediction network on the sampled volumetric features from NeRF. Our method is also one of the first to achieve such results in pure inference.
  arXiv  Detail & Related papers  (2023-04-10T05:49:24Z)
• FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization [32.1581416980828]
  We present Frequency regularized NeRF (FreeNeRF), a surprisingly simple baseline that outperforms previous methods. We analyze the key challenges in few-shot neural rendering and find that frequency plays an important role in NeRF's training.
  arXiv  Detail & Related papers  (2023-03-13T18:59:03Z)
• Self-NeRF: A Self-Training Pipeline for Few-Shot Neural Radiance Fields [17.725937326348994]
  We propose Self-NeRF, a self-evolved NeRF that iteratively refines the radiance fields with very few number of input views. In each iteration, we label unseen views with the predicted colors or warped pixels generated by the model from the preceding iteration. These expanded pseudo-views are afflicted by imprecision in color and warping artifacts, which degrades the performance of NeRF.
  arXiv  Detail & Related papers  (2023-03-10T08:22:36Z)
• ActiveNeRF: Learning where to See with Uncertainty Estimation [36.209200774203005]
  Recently, Neural Radiance Fields (NeRF) has shown promising performances on reconstructing 3D scenes and synthesizing novel views from a sparse set of 2D images. We present a novel learning framework, ActiveNeRF, aiming to model a 3D scene with a constrained input budget.
  arXiv  Detail & Related papers  (2022-09-18T12:09:15Z)
• SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image [85.43496313628943]
  We present a Single View NeRF (SinNeRF) framework consisting of thoughtfully designed semantic and geometry regularizations. SinNeRF constructs a semi-supervised learning process, where we introduce and propagate geometry pseudo labels. Experiments are conducted on complex scene benchmarks, including NeRF synthetic dataset, Local Light Field Fusion dataset, and DTU dataset.
  arXiv  Detail & Related papers  (2022-04-02T19:32:42Z)
• Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs [54.41204057689033]
  We explore how to leverage neural fields (NeRFs) to build interactive 3D environments from large-scale visual captures spanning buildings or even multiple city blocks collected primarily from drone data. In contrast to the single object scenes against which NeRFs have been traditionally evaluated, this setting poses multiple challenges. We introduce a simple clustering algorithm that partitions training images (or rather pixels) into different NeRF submodules that can be trained in parallel.
  arXiv  Detail & Related papers  (2021-12-20T17:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.