SofGAN: A Portrait Image Generator with Dynamic Styling
- URL: http://arxiv.org/abs/2007.03780v2
- Date: Fri, 6 Aug 2021 09:22:30 GMT
- Title: SofGAN: A Portrait Image Generator with Dynamic Styling
- Authors: Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, Jingyi Yu
- Abstract summary: Generative Adversarial Networks (GANs) have been widely used for portrait image generation.
We propose a SofGAN image generator to decouple the latent space of portraits into two subspaces.
We show that our system can generate high quality portrait images with independently controllable geometry and texture attributes.
- Score: 47.10046693844792
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, Generative Adversarial Networks (GANs)} have been widely used for
portrait image generation. However, in the latent space learned by GANs,
different attributes, such as pose, shape, and texture style, are generally
entangled, making the explicit control of specific attributes difficult. To
address this issue, we propose a SofGAN image generator to decouple the latent
space of portraits into two subspaces: a geometry space and a texture space.
The latent codes sampled from the two subspaces are fed to two network branches
separately, one to generate the 3D geometry of portraits with canonical pose,
and the other to generate textures. The aligned 3D geometries also come with
semantic part segmentation, encoded as a semantic occupancy field (SOF). The
SOF allows the rendering of consistent 2D semantic segmentation maps at
arbitrary views, which are then fused with the generated texture maps and
stylized to a portrait photo using our semantic instance-wise (SIW) module.
Through extensive experiments, we show that our system can generate high
quality portrait images with independently controllable geometry and texture
attributes. The method also generalizes well in various applications such as
appearance-consistent facial animation and dynamic styling.
Related papers
- FDGaussian: Fast Gaussian Splatting from Single Image via Geometric-aware Diffusion Model [81.03553265684184]
We introduce FDGaussian, a novel two-stage framework for single-image 3D reconstruction.
Recent methods typically utilize pre-trained 2D diffusion models to generate plausible novel views from the input image.
We demonstrate that FDGaussian generates images with high consistency across different views and reconstructs high-quality 3D objects.
arXiv Detail & Related papers (2024-03-15T12:24:36Z) - Pixel2ISDF: Implicit Signed Distance Fields based Human Body Model from
Multi-view and Multi-pose Images [67.45882013828256]
We focus on reconstructing clothed humans in the canonical space given multiple views and poses of a human as the input.
We learn latent codes on the posed mesh by leveraging multiple input images and then assign the latent codes to the mesh in the canonical space.
Our work for reconstructing the human shape on canonical pose achieves 3rd performance on WCPA MVP-Human Body Challenge.
arXiv Detail & Related papers (2022-12-06T05:30:49Z) - LatentSwap3D: Semantic Edits on 3D Image GANs [48.1271011449138]
3D GANs can generate latent codes for entire 3D volumes rather than only 2D images.
LatentSwap3D is a semantic edit approach based on latent space discovery.
We show results on seven 3D GANs and on five datasets.
arXiv Detail & Related papers (2022-12-02T18:59:51Z) - Self-Supervised Image Representation Learning with Geometric Set
Consistency [50.12720780102395]
We propose a method for self-supervised image representation learning under the guidance of 3D geometric consistency.
Specifically, we introduce 3D geometric consistency into a contrastive learning framework to enforce the feature consistency within image views.
arXiv Detail & Related papers (2022-03-29T08:57:33Z) - FENeRF: Face Editing in Neural Radiance Fields [34.332520597067074]
We propose FENeRF, a 3D-aware generator that can produce view-consistent and locally-editable portrait images.
Our method uses two decoupled latent codes to generate corresponding facial semantics and texture in a spatial aligned 3D volume with shared geometry.
Our experiments demonstrate that FENeRF outperforms state-of-the-art methods in various face editing tasks.
arXiv Detail & Related papers (2021-11-30T15:23:08Z) - Exemplar-Based 3D Portrait Stylization [23.585334925548064]
We present the first framework for one-shot 3D portrait style transfer.
It can generate 3D face models with both the geometry exaggerated and the texture stylized.
Our method achieves robustly good results on different artistic styles and outperforms existing methods.
arXiv Detail & Related papers (2021-04-29T17:59:54Z) - Towards Realistic 3D Embedding via View Alignment [53.89445873577063]
This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically.
VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable.
arXiv Detail & Related papers (2020-07-14T14:45:00Z) - Convolutional Generation of Textured 3D Meshes [34.20939983046376]
We propose a framework that can generate triangle meshes and associated high-resolution texture maps, using only 2D supervision from single-view natural images.
A key contribution of our work is the encoding of the mesh and texture as 2D representations, which are semantically aligned and can be easily modeled by a 2D convolutional GAN.
We demonstrate the efficacy of our method on Pascal3D+ Cars and CUB, both in an unconditional setting and in settings where the model is conditioned on class labels, attributes, and text.
arXiv Detail & Related papers (2020-06-13T15:23:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.