SideGAN: 3D-Aware Generative Model for Improved Side-View Image Synthesis

• URL: http://arxiv.org/abs/2309.10388v1
• Date: Tue, 19 Sep 2023 07:38:05 GMT
• Title: SideGAN: 3D-Aware Generative Model for Improved Side-View Image Synthesis
• Authors: Kyungmin Jo, Wonjoon Jin, Jaegul Choo, Hyunjoon Lee, Sunghyun Cho
• Abstract summary: We propose SideGAN, a novel 3D GAN training method to generate photo-realistic images irrespective of the camera pose. We show that our approach enables 3D GANs to generate high-quality geometries and photo-realistic images irrespective of the camera pose.
• Score: 44.05449141767394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While recent 3D-aware generative models have shown photo-realistic image synthesis with multi-view consistency, the synthesized image quality degrades depending on the camera pose (e.g., a face with a blurry and noisy boundary at a side viewpoint). Such degradation is mainly caused by the difficulty of learning both pose consistency and photo-realism simultaneously from a dataset with heavily imbalanced poses. In this paper, we propose SideGAN, a novel 3D GAN training method to generate photo-realistic images irrespective of the camera pose, especially for faces of side-view angles. To ease the challenging problem of learning photo-realistic and pose-consistent image synthesis, we split the problem into two subproblems, each of which can be solved more easily. Specifically, we formulate the problem as a combination of two simple discrimination problems, one of which learns to discriminate whether a synthesized image looks real or not, and the other learns to discriminate whether a synthesized image agrees with the camera pose. Based on this, we propose a dual-branched discriminator with two discrimination branches. We also propose a pose-matching loss to learn the pose consistency of 3D GANs. In addition, we present a pose sampling strategy to increase learning opportunities for steep angles in a pose-imbalanced dataset. With extensive validation, we demonstrate that our approach enables 3D GANs to generate high-quality geometries and photo-realistic images irrespective of the camera pose.

Related papers

• DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis [18.64688172651478]
  We present DiffPortrait3D, a conditional diffusion model capable of synthesizing 3D-consistent photo-realistic novel views. Given a single RGB input, we aim to synthesize plausible but consistent facial details rendered from novel camera views. We demonstrate state-of-the-art results both qualitatively and quantitatively on our challenging in-the-wild and multi-view benchmarks.
  arXiv  Detail & Related papers  (2023-12-20T13:31:11Z)
• AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image Collections [78.81539337399391]
  We present an animatable 3D-aware GAN that generates portrait images with controllable facial expression, head pose, and shoulder movements. It is a generative model trained on unstructured 2D image collections without using 3D or video data. A dual-camera rendering and adversarial learning scheme is proposed to improve the quality of the generated faces.
  arXiv  Detail & Related papers  (2023-09-05T12:44:57Z)
• Learning 3D-aware Image Synthesis with Unknown Pose Distribution [68.62476998646866]
  Existing methods for 3D-aware image synthesis largely depend on the 3D pose distribution pre-estimated on the training set. This work proposes PoF3D that frees generative radiance fields from the requirements of 3D pose priors.
  arXiv  Detail & Related papers  (2023-01-18T18:47:46Z)
• Explicitly Controllable 3D-Aware Portrait Generation [42.30481422714532]
  We propose a 3D portrait generation network that produces consistent portraits according to semantic parameters regarding pose, identity, expression and lighting. Our method outperforms prior arts in extensive experiments, producing realistic portraits with vivid expression in natural lighting when viewed in free viewpoint.
  arXiv  Detail & Related papers  (2022-09-12T17:40:08Z)
• Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency [40.56510679634943]
  We propose a self-supervised training architecture by leveraging the multi-view geometry consistency. We design three novel loss functions for multi-view consistency, including the pixel consistency loss, the depth consistency loss, and the facial landmark-based epipolar loss. Our method is accurate and robust, especially under large variations of expressions, poses, and illumination conditions.
  arXiv  Detail & Related papers  (2020-07-24T12:36:09Z)
• Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement [63.853412753242615]
  Learning a good 3D human pose representation is important for human pose related tasks. We propose a novel Siamese denoising autoencoder to learn a 3D pose representation. Our approach achieves state-of-the-art performance on two inherently different tasks.
  arXiv  Detail & Related papers  (2020-07-14T14:25:22Z)
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis [43.4859484191223]
  We propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone.
  arXiv  Detail & Related papers  (2020-07-05T20:37:39Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images [61.98279201609436]
  In this paper, we explore a more realistic setup of learning 3D shape from only single-view images. The major difficulty lies in insufficient constraints that can be provided by single view images. We propose an effective adversarial domain confusion method to learn pose-disentangled compact shape space.
  arXiv  Detail & Related papers  (2020-04-03T02:47:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.