S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch

• URL: http://arxiv.org/abs/2408.01218v1
• Date: Fri, 2 Aug 2024 12:16:07 GMT
• Title: S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch
• Authors: Zidu Wang, Xiangyu Zhu, Jiang Yu, Tianshuo Zhang, Zhen Lei,
• Abstract summary: 3D textured face reconstruction from sketches applicable in many scenarios such as animation, 3D avatars, artistic design, missing people search, etc. This paper proposes a novel method for reconstructing controllable textured and detailed 3D faces from sketches, named S2TD-Face.
• Score: 29.068915907911432
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: 3D textured face reconstruction from sketches applicable in many scenarios such as animation, 3D avatars, artistic design, missing people search, etc., is a highly promising but underdeveloped research topic. On the one hand, the stylistic diversity of sketches leads to existing sketch-to-3D-face methods only being able to handle pose-limited and realistically shaded sketches. On the other hand, texture plays a vital role in representing facial appearance, yet sketches lack this information, necessitating additional texture control in the reconstruction process. This paper proposes a novel method for reconstructing controllable textured and detailed 3D faces from sketches, named S2TD-Face. S2TD-Face introduces a two-stage geometry reconstruction framework that directly reconstructs detailed geometry from the input sketch. To keep geometry consistent with the delicate strokes of the sketch, we propose a novel sketch-to-geometry loss that ensures the reconstruction accurately fits the input features like dimples and wrinkles. Our training strategies do not rely on hard-to-obtain 3D face scanning data or labor-intensive hand-drawn sketches. Furthermore, S2TD-Face introduces a texture control module utilizing text prompts to select the most suitable textures from a library and seamlessly integrate them into the geometry, resulting in a 3D detailed face with controllable texture. S2TD-Face surpasses existing state-of-the-art methods in extensive quantitative and qualitative experiments. Our project is available at https://github.com/wang-zidu/S2TD-Face .

Related papers

• Control3D: Towards Controllable Text-to-3D Generation [107.81136630589263]
  We present a text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D. A 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF. We exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene.
  arXiv  Detail & Related papers  (2023-11-09T15:50:32Z)
• Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation [25.293899668984018]
  Facial sketch synthesis (FSS) aims to generate a vivid sketch portrait from a given facial photo. In this paper, we propose a novel Human-Inspired Dynamic Adaptation (HIDA) method. We show that HIDA can generate high-quality sketches in multiple styles, and significantly outperforms previous methods.
  arXiv  Detail & Related papers  (2023-09-01T02:27:05Z)
• Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images [105.92311979305065]
  TG-3DFace creates more realistic and aesthetically pleasing 3D faces, boosting 9% multi-view consistency (MVIC) over Latent3D. The rendered face images generated by TG-3DFace achieve higher FID and CLIP score than text-to-2D face/image generation models.
  arXiv  Detail & Related papers  (2023-08-31T14:26:33Z)
• SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling [69.28254439393298]
  SketchMetaFace is a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We develop a novel learning-based method termed "Implicit and Depth Guided Mesh Modeling" (IDGMM) It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency.
  arXiv  Detail & Related papers  (2023-07-03T07:41:07Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• SimpModeling: Sketching Implicit Field to Guide Mesh Modeling for 3D Animalmorphic Head Design [40.821865912127635]
  We propose SimpModeling, a novel sketch-based system for helping users, especially amateur users, easily model 3D animalmorphic heads. We use the advanced implicit-based shape inference methods, which have strong ability to handle the domain gap between freehand sketches and synthetic ones used for training. We also contribute to a dataset of high-quality 3D animal heads, which are manually created by artists.
  arXiv  Detail & Related papers  (2021-08-05T12:17:36Z)
• Exemplar-Based 3D Portrait Stylization [23.585334925548064]
  We present the first framework for one-shot 3D portrait style transfer. It can generate 3D face models with both the geometry exaggerated and the texture stylized. Our method achieves robustly good results on different artistic styles and outperforms existing methods.
  arXiv  Detail & Related papers  (2021-04-29T17:59:54Z)
• Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches [65.96417928860039]
  We use an encoder/decoder architecture for the sketch to mesh translation. We will show that this approach is easy to deploy, robust to style changes, and effective.
  arXiv  Detail & Related papers  (2021-04-01T14:10:59Z)
• DeepFacePencil: Creating Face Images from Freehand Sketches [77.00929179469559]
  Existing image-to-image translation methods require a large-scale dataset of paired sketches and images for supervision. We propose DeepFacePencil, an effective tool that is able to generate photo-realistic face images from hand-drawn sketches.
  arXiv  Detail & Related papers  (2020-08-31T03:35:21Z)
• 3D Shape Reconstruction from Free-Hand Sketches [42.15888734492648]
  Despite great progress achieved in 3D reconstruction from distortion-free line drawings, little effort has been made to reconstruct 3D shapes from free-hand sketches. We aim to enhance the power of sketches in 3D-related applications such as interactive design and VR/AR games. A major challenge for free-hand sketch 3D reconstruction comes from the insufficient training data and free-hand sketch diversity.
  arXiv  Detail & Related papers  (2020-06-17T07:43:10Z)
• Landmark Detection and 3D Face Reconstruction for Caricature using a Nonlinear Parametric Model [27.553158595012974]
  We propose the first automatic method for automatic landmark detection and 3D face reconstruction for caricature. Based on the constructed dataset and the nonlinear parametric model, we propose a neural network based method to regress the 3D face shape and orientation from the input 2D caricature image.
  arXiv  Detail & Related papers  (2020-04-20T10:34:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.