CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis

• URL: http://arxiv.org/abs/2110.09788v1
• Date: Tue, 19 Oct 2021 08:02:16 GMT
• Title: CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
• Authors: Peng Zhou, Lingxi Xie, Bingbing Ni, Qi Tian
• Abstract summary: This paper presents CIPS-3D, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation network. The generator synthesizes each pixel value independently without any spatial convolution or upsampling operation. It sets new records for 3D-aware image synthesis with an impressive FID of 6.97 for images at the $256times256$ resolution on FFHQ.
• Score: 148.4104739574094
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The style-based GAN (StyleGAN) architecture achieved state-of-the-art results for generating high-quality images, but it lacks explicit and precise control over camera poses. The recently proposed NeRF-based GANs made great progress towards 3D-aware generators, but they are unable to generate high-quality images yet. This paper presents CIPS-3D, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation (INR) network. The generator synthesizes each pixel value independently without any spatial convolution or upsampling operation. In addition, we diagnose the problem of mirror symmetry that implies a suboptimal solution and solve it by introducing an auxiliary discriminator. Trained on raw, single-view images, CIPS-3D sets new records for 3D-aware image synthesis with an impressive FID of 6.97 for images at the $256\times256$ resolution on FFHQ. We also demonstrate several interesting directions for CIPS-3D such as transfer learning and 3D-aware face stylization. The synthesis results are best viewed as videos, so we recommend the readers to check our github project at https://github.com/PeterouZh/CIPS-3D

Related papers

• IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation [96.32684334038278]
  In this paper, we explore the design space of text-to-3D models. We significantly improve multi-view generation by considering video instead of image generators. Our new method, IM-3D, reduces the number of evaluations of the 2D generator network 10-100x.
  arXiv  Detail & Related papers  (2024-02-13T18:59:51Z)
• Free3D: Consistent Novel View Synthesis without 3D Representation [63.931920010054064]
  Free3D is a simple accurate method for monocular open-set novel view synthesis (NVS) Compared to other works that took a similar approach, we obtain significant improvements without resorting to an explicit 3D representation.
  arXiv  Detail & Related papers  (2023-12-07T18:59:18Z)
• WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space [77.92350895927922]
  We propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs) Our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data.
  arXiv  Detail & Related papers  (2023-11-22T18:25:51Z)
• Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation [29.959223778769513]
  We propose a novel learning strategy, namely 3D-to-2D imitation, which enables a 3D-aware GAN to generate high-quality images. We also introduce 3D-aware convolutions into the generator for better 3D representation learning. With the above strategies, our method reaches FID scores of 5.4 and 4.3 on FFHQ and AFHQ-v2 Cats, respectively, at 512x512 resolution.
  arXiv  Detail & Related papers  (2023-03-16T02:18:41Z)
• Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator [68.0533826852601]
  3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. Existing methods fail to obtain moderate 3D shapes. We propose a geometry-aware discriminator to improve 3D-aware GANs.
  arXiv  Detail & Related papers  (2022-09-30T17:59:37Z)
• EpiGRAF: Rethinking training of 3D GANs [60.38818140637367]
  We show that it is possible to obtain a high-resolution 3D generator with SotA image quality by following a completely different route of simply training the model patch-wise. The resulting model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator.
  arXiv  Detail & Related papers  (2022-06-21T17:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.