ZIGNeRF: Zero-shot 3D Scene Representation with Invertible Generative Neural Radiance Fields

• URL: http://arxiv.org/abs/2306.02741v1
• Date: Mon, 5 Jun 2023 09:41:51 GMT
• Title: ZIGNeRF: Zero-shot 3D Scene Representation with Invertible Generative Neural Radiance Fields
• Authors: Kanghyeok Ko, Minhyeok Lee
• Abstract summary: We introduce ZIGNeRF, an innovative model that executes zero-shot Generative Adrial Network (GAN)versa for the generation of multi-view images from a single out-of-domain image. ZIGNeRF is capable of disentangling the object from the background and executing 3D operations such as 360-degree rotation or depth and horizontal translation.
• Score: 2.458437232470188
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative Neural Radiance Fields (NeRFs) have demonstrated remarkable proficiency in synthesizing multi-view images by learning the distribution of a set of unposed images. Despite the aptitude of existing generative NeRFs in generating 3D-consistent high-quality random samples within data distribution, the creation of a 3D representation of a singular input image remains a formidable challenge. In this manuscript, we introduce ZIGNeRF, an innovative model that executes zero-shot Generative Adversarial Network (GAN) inversion for the generation of multi-view images from a single out-of-domain image. The model is underpinned by a novel inverter that maps out-of-domain images into the latent code of the generator manifold. Notably, ZIGNeRF is capable of disentangling the object from the background and executing 3D operations such as 360-degree rotation or depth and horizontal translation. The efficacy of our model is validated using multiple real-image datasets: Cats, AFHQ, CelebA, CelebA-HQ, and CompCars.

Related papers

• NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model [57.92709692193132]
  NovelGS is a diffusion model for Gaussian Splatting given sparse-view images. We leverage the novel view denoising through a transformer-based network to generate 3D Gaussians.
  arXiv  Detail & Related papers  (2024-11-25T07:57:17Z)
• GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise [0.0]
  This paper introduces a novel text to 3D content generation framework based on Gaussian splatting. The challenge of achieving multi-view consistency in 3D generation significantly impedes modeling complexity and accuracy.
  arXiv  Detail & Related papers  (2023-11-19T04:26:16Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• NeRF-GAN Distillation for Efficient 3D-Aware Generation with Convolutions [97.27105725738016]
  integration of Neural Radiance Fields (NeRFs) and generative models, such as Generative Adversarial Networks (GANs) has transformed 3D-aware generation from single-view images. We propose a simple and effective method, based on re-using the well-disentangled latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional network to directly generate 3D-consistent images corresponding to the underlying 3D representations.
  arXiv  Detail & Related papers  (2023-03-22T18:59:48Z)
• NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion [107.67277084886929]
  Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input. We propose NerfDiff, which addresses this issue by distilling the knowledge of a 3D-aware conditional diffusion model (CDM) into NeRF through synthesizing and refining a set of virtual views at test time. We further propose a novel NeRF-guided distillation algorithm that simultaneously generates 3D consistent virtual views from the CDM samples, and finetunes the NeRF based on the improved virtual views.
  arXiv  Detail & Related papers  (2023-02-20T17:12:00Z)
• NeRFInvertor: High Fidelity NeRF-GAN Inversion for Single-shot Real Image Animation [66.0838349951456]
  Nerf-based Generative models have shown impressive capacity in generating high-quality images with consistent 3D geometry. We propose a universal method to surgically fine-tune these NeRF-GAN models in order to achieve high-fidelity animation of real subjects only by a single image.
  arXiv  Detail & Related papers  (2022-11-30T18:36:45Z)
• Pix2NeRF: Unsupervised Conditional $\pi$-GAN for Single Image to Neural Radiance Fields Translation [93.77693306391059]
  We propose a pipeline to generate Neural Radiance Fields(NeRF) of an object or a scene of a specific class, conditioned on a single input image. Our method is based on $pi$-GAN, a generative model for unconditional 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2022-02-26T15:28:05Z)
• GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation [25.20217335614512]
  3D-aware image generative modeling aims to generate 3D-consistent images with explicitly controllable camera poses. Recent works have shown promising results by training neural radiance field (NeRF) generators on unstructured 2D images.
  arXiv  Detail & Related papers  (2021-12-16T13:25:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.