HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks

• URL: http://arxiv.org/abs/2304.09463v1
• Date: Wed, 19 Apr 2023 07:22:05 GMT
• Title: HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks
• Authors: Zhuo Chen, Xudong Xu, Yichao Yan, Ye Pan, Wenhan Zhu, Wayne Wu, Bo Dai and Xiaokang Yang
• Abstract summary: This paper is inspired by the success of 3D-aware GANs that bridge 2D and 3D domains with 3D fields as the intermediate representation for rendering 2D images. We propose a novel method, dubbed HyperStyle3D, based on 3D-aware GANs for 3D portrait stylization.
• Score: 101.36230756743106
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Portrait stylization is a long-standing task enabling extensive applications. Although 2D-based methods have made great progress in recent years, real-world applications such as metaverse and games often demand 3D content. On the other hand, the requirement of 3D data, which is costly to acquire, significantly impedes the development of 3D portrait stylization methods. In this paper, inspired by the success of 3D-aware GANs that bridge 2D and 3D domains with 3D fields as the intermediate representation for rendering 2D images, we propose a novel method, dubbed HyperStyle3D, based on 3D-aware GANs for 3D portrait stylization. At the core of our method is a hyper-network learned to manipulate the parameters of the generator in a single forward pass. It not only offers a strong capacity to handle multiple styles with a single model, but also enables flexible fine-grained stylization that affects only texture, shape, or local part of the portrait. While the use of 3D-aware GANs bypasses the requirement of 3D data, we further alleviate the necessity of style images with the CLIP model being the stylization guidance. We conduct an extensive set of experiments across the style, attribute, and shape, and meanwhile, measure the 3D consistency. These experiments demonstrate the superior capability of our HyperStyle3D model in rendering 3D-consistent images in diverse styles, deforming the face shape, and editing various attributes.

Related papers

• Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering [17.918603435615335]
  3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. We propose Diff3DS, a novel differentiable framework for generating view-consistent 3D sketch. Our framework bridges the domains of 3D sketch and customized image, achieving end-toend optimization of 3D sketch.
  arXiv  Detail & Related papers  (2024-05-24T07:48:14Z)
• Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior [52.44678180286886]
  2D diffusion models find a distillation approach that achieves excellent generalization and rich details without any 3D data. We propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity, generalizability, and geometric consistency simultaneously.
  arXiv  Detail & Related papers  (2023-12-11T18:59:18Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• Freestyle 3D-Aware Portrait Synthesis Based on Compositional Generative Priors [12.663585627797863]
  We propose a novel text-driven 3D-aware portrait synthesis framework. Specifically, for a given portrait style prompt, we first composite two generative priors, a 3D-aware GAN generator and a text-guided image editor. Then we map the special style domain of this set to our proposed 3D latent feature generator and obtain a 3D representation containing the given style information.
  arXiv  Detail & Related papers  (2023-06-27T12:23:04Z)
• AG3D: Learning to Generate 3D Avatars from 2D Image Collections [96.28021214088746]
  We propose a new adversarial generative model of realistic 3D people from 2D images. Our method captures shape and deformation of the body and loose clothing by adopting a holistic 3D generator. We experimentally find that our method outperforms previous 3D- and articulation-aware methods in terms of geometry and appearance.
  arXiv  Detail & Related papers  (2023-05-03T17:56:24Z)
• AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning [80.67196184480754]
  We propose a novel framework emphAgileGAN3D that can produce 3D artistically appealing portraits with detailed geometry. New stylization can be obtained with just a few (around 20) unpaired 2D exemplars. Our pipeline demonstrates strong capability in turning user photos into a diverse range of 3D artistic portraits.
  arXiv  Detail & Related papers  (2023-03-24T23:04:20Z)
• XDGAN: Multi-Modal 3D Shape Generation in 2D Space [60.46777591995821]
  We propose a novel method to convert 3D shapes into compact 1-channel geometry images and leverage StyleGAN3 and image-to-image translation networks to generate 3D objects in 2D space. The generated geometry images are quick to convert to 3D meshes, enabling real-time 3D object synthesis, visualization and interactive editing. We show both quantitatively and qualitatively that our method is highly effective at various tasks such as 3D shape generation, single view reconstruction and shape manipulation, while being significantly faster and more flexible compared to recent 3D generative models.
  arXiv  Detail & Related papers  (2022-10-06T15:54:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.