Text-Guided 3D Face Synthesis -- From Generation to Editing

• URL: http://arxiv.org/abs/2312.00375v1
• Date: Fri, 1 Dec 2023 06:36:23 GMT
• Title: Text-Guided 3D Face Synthesis -- From Generation to Editing
• Authors: Yunjie Wu, Yapeng Meng, Zhipeng Hu, Lincheng Li, Haoqian Wu, Kun Zhou, Weiwei Xu, Xin Yu
• Abstract summary: We propose a unified text-guided framework from face generation to editing. We employ a fine-tuned texture diffusion model to enhance texture quality in both RGB and YUV space. We propose a self-guided consistency weight strategy to improve editing efficacy while preserving consistency.
• Score: 53.86765812392627
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-guided 3D face synthesis has achieved remarkable results by leveraging text-to-image (T2I) diffusion models. However, most existing works focus solely on the direct generation, ignoring the editing, restricting them from synthesizing customized 3D faces through iterative adjustments. In this paper, we propose a unified text-guided framework from face generation to editing. In the generation stage, we propose a geometry-texture decoupled generation to mitigate the loss of geometric details caused by coupling. Besides, decoupling enables us to utilize the generated geometry as a condition for texture generation, yielding highly geometry-texture aligned results. We further employ a fine-tuned texture diffusion model to enhance texture quality in both RGB and YUV space. In the editing stage, we first employ a pre-trained diffusion model to update facial geometry or texture based on the texts. To enable sequential editing, we introduce a UV domain consistency preservation regularization, preventing unintentional changes to irrelevant facial attributes. Besides, we propose a self-guided consistency weight strategy to improve editing efficacy while preserving consistency. Through comprehensive experiments, we showcase our method's superiority in face synthesis. Project page: https://faceg2e.github.io/.

Related papers

• MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing [61.014328598895524]
  We propose textbfMaTe3D: mask-guided text-based 3D-aware portrait editing. New SDF-based 3D generator learns local and global representations with proposed SDF and density consistency losses. Conditional Distillation on Geometry and Texture (CDGT) mitigates visual ambiguity and avoids mismatch between texture and geometry.
  arXiv  Detail & Related papers  (2023-12-12T03:04:08Z)
• TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models [77.85129451435704]
  We present a new method to synthesize textures for 3D, using large-scale-guided image diffusion models. Specifically, we leverage latent diffusion models, apply the set denoising model and aggregate denoising text map.
  arXiv  Detail & Related papers  (2023-10-20T19:15:29Z)
• Directional Texture Editing for 3D Models [51.31499400557996]
  ITEM3D is designed for automatic textbf3D object editing according to the text textbfInstructions. Leveraging the diffusion models and the differentiable rendering, ITEM3D takes the rendered images as the bridge of text and 3D representation.
  arXiv  Detail & Related papers  (2023-09-26T12:01:13Z)
• Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images [105.92311979305065]
  TG-3DFace creates more realistic and aesthetically pleasing 3D faces, boosting 9% multi-view consistency (MVIC) over Latent3D. The rendered face images generated by TG-3DFace achieve higher FID and CLIP score than text-to-2D face/image generation models.
  arXiv  Detail & Related papers  (2023-08-31T14:26:33Z)
• Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion [115.82306502822412]
  StyleGAN has achieved great progress in 2D face reconstruction and semantic editing via image inversion and latent editing. A corresponding generic 3D GAN inversion framework is still missing, limiting the applications of 3D face reconstruction and semantic editing. We study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures.
  arXiv  Detail & Related papers  (2022-12-14T18:49:50Z)
• ClipFace: Text-guided Editing of Textured 3D Morphable Models [33.83015491013442]
  We propose ClipFace, a novel self-supervised approach for text-guided editing of textured 3D morphable model of faces. We employ user-friendly language prompts to enable control of the expressions as well as appearance of 3D faces. Our model is trained in a self-supervised fashion by exploiting differentiable rendering and losses based on a pre-trained CLIP model.
  arXiv  Detail & Related papers  (2022-12-02T19:01:08Z)
• NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields [40.543998582101146]
  We introduce NeRFFaceEditing, which enables editing and decoupling geometry and appearance in neural radiance fields. Our method allows users to edit via semantic masks with decoupled control of geometry and appearance. Both qualitative and quantitative evaluations show the superior geometry and appearance control abilities of our method compared to existing and alternative solutions.
  arXiv  Detail & Related papers  (2022-11-15T08:11:39Z)
• NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing [39.71252429542249]
  We present a novel mesh-based representation by encoding the neural implicit field with disentangled geometry and texture codes on mesh vertices. We develop several techniques including learnable sign indicators to magnify spatial distinguishability of mesh-based representation. Experiments and editing examples on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability.
  arXiv  Detail & Related papers  (2022-07-25T05:30:50Z)
• Neural Parameterization for Dynamic Human Head Editing [26.071370285285465]
  We present Neuralization (NeP), a hybrid representation that provides the advantages of both implicit and explicit methods. NeP is capable of photo-realistic rendering while allowing fine-grained editing of the scene geometry and appearance. The results show that the NeP achieves almost the same level of rendering accuracy while maintaining high editability.
  arXiv  Detail & Related papers  (2022-07-01T05:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.