GAN2X: Non-Lambertian Inverse Rendering of Image GANs

• URL: http://arxiv.org/abs/2206.09244v1
• Date: Sat, 18 Jun 2022 16:58:49 GMT
• Title: GAN2X: Non-Lambertian Inverse Rendering of Image GANs
• Authors: Xingang Pan, Ayush Tewari, Lingjie Liu, Christian Theobalt
• Abstract summary: We present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also recover non-Lambertian material properties by exploiting the pseudo paired data generated by a GAN. Experiments demonstrate that GAN2X can accurately decompose 2D images to 3D shape, albedo, and specular properties for different object categories, and achieves the state-of-the-art performance for unsupervised single-view 3D face reconstruction.
• Score: 85.76426471872855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components. Recovering these underlying intrinsic components from 2D images, also known as inverse rendering, usually requires a supervised setting with paired images collected from multiple viewpoints and lighting conditions, which is resource-demanding. In this work, we present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also recover non-Lambertian material properties by exploiting the pseudo paired data generated by a GAN. To achieve precise inverse rendering, we devise a specularity-aware neural surface representation that continuously models the geometry and material properties. A shading-based refinement technique is adopted to further distill information in the target image and recover more fine details. Experiments demonstrate that GAN2X can accurately decompose 2D images to 3D shape, albedo, and specular properties for different object categories, and achieves the state-of-the-art performance for unsupervised single-view 3D face reconstruction. We also show its applications in downstream tasks including real image editing and lifting 2D GANs to decomposed 3D GANs.

Related papers

• Unsupervised Style-based Explicit 3D Face Reconstruction from Single Image [10.1205208477163]
  In this work, we propose a general adversarial learning framework for solving Unsupervised 2D to Explicit 3D Style Transfer. Specifically, we merge two architectures: the unsupervised explicit 3D reconstruction network of Wu et al. and the Generative Adversarial Network (GAN) named StarGAN-v2. We show that our solution is able to outperform well established solutions such as DepthNet in 3D reconstruction and Pix2NeRF in conditional style transfer.
  arXiv  Detail & Related papers  (2023-04-24T21:25:06Z)
• In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing [28.790900756506833]
  3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs.
  arXiv  Detail & Related papers  (2023-02-09T18:59:56Z)
• Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion [115.82306502822412]
  StyleGAN has achieved great progress in 2D face reconstruction and semantic editing via image inversion and latent editing. A corresponding generic 3D GAN inversion framework is still missing, limiting the applications of 3D face reconstruction and semantic editing. We study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures.
  arXiv  Detail & Related papers  (2022-12-14T18:49:50Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction [33.95791350070165]
  Inferring 3D structure of a generic object from a 2D image is a long-standing objective of computer vision. We take an alternative approach with semi-supervised learning. That is, for a 2D image of a generic object, we decompose it into latent representations of category, shape and albedo. We show that the complete shape and albedo modeling enables us to leverage real 2D images in both modeling and model fitting.
  arXiv  Detail & Related papers  (2021-04-02T02:39:29Z)
• Lifting 2D StyleGAN for 3D-Aware Face Generation [52.8152883980813]
  We propose a framework, called LiftedGAN, that disentangles and lifts a pre-trained StyleGAN2 for 3D-aware face generation. Our model is "3D-aware" in the sense that it is able to (1) disentangle the latent space of StyleGAN2 into texture, shape, viewpoint, lighting and (2) generate 3D components for synthetic images.
  arXiv  Detail & Related papers  (2020-11-26T05:02:09Z)
• Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs [156.1209884183522]
  State-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold. We present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only.
  arXiv  Detail & Related papers  (2020-11-02T09:38:43Z)
• Towards Realistic 3D Embedding via View Alignment [53.89445873577063]
  This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically. VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable.
  arXiv  Detail & Related papers  (2020-07-14T14:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.