Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images

• URL: http://arxiv.org/abs/2409.20530v1
• Date: Mon, 30 Sep 2024 17:30:23 GMT
• Title: Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images
• Authors: Bahri Batuhan Bilecen, Ahmet Berke Gokmen, Aysegul Dundar,
• Abstract summary: 3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN) There exist encoders that achieve good results in 3D GAN inversion, but they are predominantly built on EG3D. We propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective.
• Score: 8.558093666229553
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: 3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results in 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and is limiting in synthesizing comprehensive 3D scenes from diverse viewpoints. In contrast to existing approaches, we propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective. To achieve realistic 3D modeling of the input image, we introduce a dual encoder system tailored for high-fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, we propose a stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively. Please visit the project page: https://berkegokmen1.github.io/dual-enc-3d-gan-inv.

Related papers

• GOEmbed: Gradient Origin Embeddings for Representation Agnostic 3D Feature Learning [67.61509647032862]
  We propose GOEmbed (Gradient Origin Embeddings) that encodes input 2D images into any 3D representation. Unlike typical prior approaches in which input images are encoded using 2D features extracted from large pre-trained models, or customized features are designed to handle different 3D representations.
  arXiv  Detail & Related papers  (2023-12-14T08:39:39Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [114.47216525866435]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• TriPlaneNet: An Encoder for EG3D Inversion [1.9567015559455132]
  NeRF-based GANs have introduced a number of approaches for high-resolution and high-fidelity generative modeling of human heads. Despite the success of universal optimization-based methods for 2D GAN inversion, those applied to 3D GANs may fail to extrapolate the result onto the novel view. We introduce a fast technique that bridges the gap between the two approaches by directly utilizing the tri-plane representation presented for the EG3D generative model.
  arXiv  Detail & Related papers  (2023-03-23T17:56:20Z)
• Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encoding [25.86312557482366]
  3D GAN inversion aims to achieve high reconstruction fidelity and reasonable 3D geometry simultaneously from a single image input. We introduce a novel encoder-based inversion framework based on EG3D, one of the most widely-used 3D GAN models. Our method achieves impressive results comparable to optimization-based methods while operating up to 500 times faster.
  arXiv  Detail & Related papers  (2023-03-22T05:51:53Z)
• CC3D: Layout-Conditioned Generation of Compositional 3D Scenes [49.281006972028194]
  We introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts. Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality.
  arXiv  Detail & Related papers  (2023-03-21T17:59:02Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• 3D GAN Inversion with Pose Optimization [26.140281977885376]
  We introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing.
  arXiv  Detail & Related papers  (2022-10-13T19:06:58Z)
• Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator [68.0533826852601]
  3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. Existing methods fail to obtain moderate 3D shapes. We propose a geometry-aware discriminator to improve 3D-aware GANs.
  arXiv  Detail & Related papers  (2022-09-30T17:59:37Z)
• 3D-Aware Indoor Scene Synthesis with Depth Priors [62.82867334012399]
  Existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside. We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry.
  arXiv  Detail & Related papers  (2022-02-17T09:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.