Generative Deformable Radiance Fields for Disentangled Image Synthesis of Topology-Varying Objects

• URL: http://arxiv.org/abs/2209.04183v1
• Date: Fri, 9 Sep 2022 08:44:06 GMT
• Title: Generative Deformable Radiance Fields for Disentangled Image Synthesis of Topology-Varying Objects
• Authors: Ziyu Wang, Yu Deng, Jiaolong Yang, Jingyi Yu, Xin Tong
• Abstract summary: 3D-aware generative models have demonstrated their superb performance to generate 3D neural radiance fields (NeRF) from a collection of monocular 2D images. We propose a generative model for synthesizing radiance fields of topology-varying objects with disentangled shape and appearance variations.
• Score: 52.46838926521572
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D-aware generative models have demonstrated their superb performance to generate 3D neural radiance fields (NeRF) from a collection of monocular 2D images even for topology-varying object categories. However, these methods still lack the capability to separately control the shape and appearance of the objects in the generated radiance fields. In this paper, we propose a generative model for synthesizing radiance fields of topology-varying objects with disentangled shape and appearance variations. Our method generates deformable radiance fields, which builds the dense correspondence between the density fields of the objects and encodes their appearances in a shared template field. Our disentanglement is achieved in an unsupervised manner without introducing extra labels to previous 3D-aware GAN training. We also develop an effective image inversion scheme for reconstructing the radiance field of an object in a real monocular image and manipulating its shape and appearance. Experiments show that our method can successfully learn the generative model from unstructured monocular images and well disentangle the shape and appearance for objects (e.g., chairs) with large topological variance. The model trained on synthetic data can faithfully reconstruct the real object in a given single image and achieve high-quality texture and shape editing results.

Related papers

• Object-Centric Domain Randomization for 3D Shape Reconstruction in the Wild [22.82439286651921]
  One of the biggest challenges in single-view 3D shape reconstruction in the wild is the scarcity of 3D shape, 2D image>-paired data from real-world environments. Inspired by remarkable achievements via domain randomization, we propose ObjectDR which synthesizes such paired data via a random simulation of visual variations in object appearances and backgrounds.
  arXiv  Detail & Related papers  (2024-03-21T16:40:10Z)
• SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model [63.685132323224124]
  Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains. In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges. Experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.
  arXiv  Detail & Related papers  (2024-03-15T06:26:46Z)
• Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion [54.151979979158085]
  We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available. We leverage an unconditional 3D-aware generator, to which we apply a hybrid inversion scheme where a model produces a first guess of the solution. Our framework can de-render an image in as few as 10 steps, enabling its use in practical scenarios.
  arXiv  Detail & Related papers  (2022-11-21T17:42:42Z)
• Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images [94.49117671450531]
  State-of-the-art 3D generative models are GANs which use neural 3D volumetric representations for synthesis. In this paper, we design a 3D GAN which can learn a disentangled model of objects, just from monocular observations.
  arXiv  Detail & Related papers  (2022-03-29T22:03:18Z)
• LOLNeRF: Learn from One Look [22.771493686755544]
  We present a method for learning a generative 3D model based on neural radiance fields. We show that, unlike existing methods, one does not need multi-view data to achieve this goal.
  arXiv  Detail & Related papers  (2021-11-19T01:20:01Z)
• Learning Canonical 3D Object Representation for Fine-Grained Recognition [77.33501114409036]
  We propose a novel framework for fine-grained object recognition that learns to recover object variation in 3D space from a single image. We represent an object as a composition of 3D shape and its appearance, while eliminating the effect of camera viewpoint. By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object.
  arXiv  Detail & Related papers  (2021-08-10T12:19:34Z)
• FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling [11.432178728985956]
  We use Neural Radiance Fields (NeRF) to learn high quality 3D object category models from collections of input images. We show that this method can learn accurate 3D object category models using only photometric supervision and casually captured images.
  arXiv  Detail & Related papers  (2021-04-17T01:38:54Z)
• ShaRF: Shape-conditioned Radiance Fields from a Single View [54.39347002226309]
  We present a method for estimating neural scenes representations of objects given only a single image. The core of our method is the estimation of a geometric scaffold for the object. We demonstrate in several experiments the effectiveness of our approach in both synthetic and real images.
  arXiv  Detail & Related papers  (2021-02-17T16:40:28Z)
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis [43.4859484191223]
  We propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone.
  arXiv  Detail & Related papers  (2020-07-05T20:37:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.