Sketch-1-to-3: One Single Sketch to 3D Detailed Face Reconstruction

• URL: http://arxiv.org/abs/2502.17852v1
• Date: Tue, 25 Feb 2025 04:58:17 GMT
• Title: Sketch-1-to-3: One Single Sketch to 3D Detailed Face Reconstruction
• Authors: Liting Wen, Zimo Yang, Xianlin Zhang, Chi Ding, Yue Zhang, Mingdao Wang, Xueming Li,
• Abstract summary: 3D face reconstruction from a single sketch is a critical yet underexplored task with significant practical applications.<n>We introduce Sketch-1-to-3, a novel framework for realistic 3D face reconstruction from a single sketch.<n>We show that Sketch-1-to-3 achieves state-of-the-art performance in sketch-based 3D face reconstruction.
• Score: 8.226562536837003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D face reconstruction from a single sketch is a critical yet underexplored task with significant practical applications. The primary challenges stem from the substantial modality gap between 2D sketches and 3D facial structures, including: (1) accurately extracting facial keypoints from 2D sketches; (2) preserving diverse facial expressions and fine-grained texture details; and (3) training a high-performing model with limited data. In this paper, we propose Sketch-1-to-3, a novel framework for realistic 3D face reconstruction from a single sketch, to address these challenges. Specifically, we first introduce the Geometric Contour and Texture Detail (GCTD) module, which enhances the extraction of geometric contours and texture details from facial sketches. Additionally, we design a deep learning architecture with a domain adaptation module and a tailored loss function to align sketches with the 3D facial space, enabling high-fidelity expression and texture reconstruction. To facilitate evaluation and further research, we construct SketchFaces, a real hand-drawn facial sketch dataset, and Syn-SketchFaces, a synthetic facial sketch dataset. Extensive experiments demonstrate that Sketch-1-to-3 achieves state-of-the-art performance in sketch-based 3D face reconstruction.

Related papers

• S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch [29.068915907911432]
  3D textured face reconstruction from sketches applicable in many scenarios such as animation, 3D avatars, artistic design, missing people search, etc. This paper proposes a novel method for reconstructing controllable textured and detailed 3D faces from sketches, named S2TD-Face.
  arXiv  Detail & Related papers  (2024-08-02T12:16:07Z)
• Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation [55.73399465968594]
  This paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss.
  arXiv  Detail & Related papers  (2024-04-02T11:03:24Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• PAniC-3D: Stylized Single-view 3D Reconstruction from Portraits of Anime Characters [29.107457721261387]
  We propose PAniC-3D, a system to reconstruct stylized 3D character heads directly from illustrated (p)ortraits of (ani)me (c)haracters.
  arXiv  Detail & Related papers  (2023-03-25T23:36:17Z)
• HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details [66.74088288846491]
  HiFace aims at high-fidelity 3D face reconstruction with dynamic and static details. We exploit several loss functions to jointly learn the coarse shape and fine details with both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-20T16:07:02Z)
• Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation [51.64832538714455]
  Existing studies only generate portraits in the 2D plane with fixed views, making the results less vivid. In this paper, we present Stereoscopic Simplified Sketch-to-Portrait (SSSP), which explores the possibility of creating Stereoscopic 3D-aware portraits. Our key insight is to design sketch-aware constraints that can fully exploit the prior knowledge of a tri-plane-based 3D-aware generative model.
  arXiv  Detail & Related papers  (2023-02-14T06:28:42Z)
• Facial Geometric Detail Recovery via Implicit Representation [147.07961322377685]
  We present a robust texture-guided geometric detail recovery approach using only a single in-the-wild facial image. Our method combines high-quality texture completion with the powerful expressiveness of implicit surfaces. Our method not only recovers accurate facial details but also decomposes normals, albedos, and shading parts in a self-supervised way.
  arXiv  Detail & Related papers  (2022-03-18T01:42:59Z)
• SimpModeling: Sketching Implicit Field to Guide Mesh Modeling for 3D Animalmorphic Head Design [40.821865912127635]
  We propose SimpModeling, a novel sketch-based system for helping users, especially amateur users, easily model 3D animalmorphic heads. We use the advanced implicit-based shape inference methods, which have strong ability to handle the domain gap between freehand sketches and synthetic ones used for training. We also contribute to a dataset of high-quality 3D animal heads, which are manually created by artists.
  arXiv  Detail & Related papers  (2021-08-05T12:17:36Z)
• Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches [65.96417928860039]
  We use an encoder/decoder architecture for the sketch to mesh translation. We will show that this approach is easy to deploy, robust to style changes, and effective.
  arXiv  Detail & Related papers  (2021-04-01T14:10:59Z)
• 3D Shape Reconstruction from Free-Hand Sketches [42.15888734492648]
  Despite great progress achieved in 3D reconstruction from distortion-free line drawings, little effort has been made to reconstruct 3D shapes from free-hand sketches. We aim to enhance the power of sketches in 3D-related applications such as interactive design and VR/AR games. A major challenge for free-hand sketch 3D reconstruction comes from the insufficient training data and free-hand sketch diversity.
  arXiv  Detail & Related papers  (2020-06-17T07:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.