SketchFaceGS: Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting

• URL: http://arxiv.org/abs/2604.19202v1
• Date: Tue, 21 Apr 2026 08:09:13 GMT
• Title: SketchFaceGS: Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting
• Abstract summary: SketchFaceGS is a sketch-driven framework for real-time generation and editing of 3D Gaussian head models from 2D sketches.<n>A UV Mask Fusion technique combined with a layer-by-layer feature-fusion strategy enables precise, real-time, free-viewpoint modifications.<n>Experiments show that SketchFaceGS outperforms existing methods in both generation fidelity and editing flexibility.
• Score: 39.956883930526416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Gaussian representations have emerged as a powerful paradigm for digital head modeling, achieving photorealistic quality with real-time rendering. However, intuitive and interactive creation or editing of 3D Gaussian head models remains challenging. Although 2D sketches provide an ideal interaction modality for fast, intuitive conceptual design, they are sparse, depth-ambiguous, and lack high-frequency appearance cues, making it difficult to infer dense, geometrically consistent 3D Gaussian structures from strokes - especially under real-time constraints. To address these challenges, we propose SketchFaceGS, the first sketch-driven framework for real-time generation and editing of photorealistic 3D Gaussian head models from 2D sketches. Our method uses a feed-forward, coarse-to-fine architecture. A Transformer-based UV feature-prediction module first reconstructs a coarse but geometrically consistent UV feature map from the input sketch, and then a 3D UV feature enhancement module refines it with high-frequency, photorealistic detail to produce a high-fidelity 3D head. For editing, we introduce a UV Mask Fusion technique combined with a layer-by-layer feature-fusion strategy, enabling precise, real-time, free-viewpoint modifications. Extensive experiments show that SketchFaceGS outperforms existing methods in both generation fidelity and editing flexibility, producing high-quality, editable 3D heads from sketches in a single forward pass.

Related papers

• ObjectMorpher: 3D-Aware Image Editing via Deformable 3DGS Models [31.723421257306384]
  We present ObjectMorpher, a unified, interactive framework that converts ambiguous 2D edits into geometry-grounded operations.<n>Users drag control points; a graph-based non-rigid deformation with as-rigid-as-possible (ARAP) constraints ensures physically sensible shape and pose changes.<n>Across diverse categories, ObjectMorpher delivers fine-grained, photorealistic edits with superior controllability and efficiency.
  arXiv  Detail & Related papers  (2026-03-30T08:15:29Z)
• PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing [51.56943889042673]
  PercHead is a method for single-image 3D head reconstruction and semantic 3D editing.<n>We develop a unified base model for reconstructing view-consistent 3D heads from a single input image.<n>We highlight the intuitive and powerful 3D editing capabilities of our model through a lightweight, interactive GUI.
  arXiv  Detail & Related papers  (2025-11-04T17:59:15Z)
• SVG-Head: Hybrid Surface-Volumetric Gaussians for High-Fidelity Head Reconstruction and Real-Time Editing [26.400512371742096]
  Surface-Volumetric Gaussian Head Avatar (SVG-Head) is a novel hybrid representation that explicitly models the geometry with 3D Gaussians bound on a FLAME mesh.<n>To model the correspondence between 3D world and texture space, we provide a mesh-aware Gaussian UV mapping method.<n>Experiments on the NeRSemble dataset show that SVG-Head not only generates high-fidelity rendering results, but also is the first method to obtain explicit texture images for Gaussian head avatars.
  arXiv  Detail & Related papers  (2025-08-13T08:27:55Z)
• From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videos [8.444819892052958]
  2D diffusion models have shown promise for producing photorealistic edits.<n>Traditional 3D rendering methods excel in spatial and temporal consistency but fall short in achieving photorealistic lighting.<n>This is the first approach to synergize 3D rendering and 2D diffusion for video object insertion.
  arXiv  Detail & Related papers  (2025-07-27T15:49:07Z)
• EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis [61.1662426227688]
  Existing NeRF and 3DGS-based methods show promising results in achieving photorealistic renderings but require slow, per-scene optimization.<n>We introduce EVolSplat, an efficient 3D Gaussian Splatting model for urban scenes that works in a feed-forward manner.
  arXiv  Detail & Related papers  (2025-03-26T02:47:27Z)
• Generating Editable Head Avatars with 3D Gaussian GANs [57.51487984425395]
  Traditional 3D-aware generative adversarial networks (GANs) achieve photorealistic and view-consistent 3D head synthesis.<n>We propose a novel approach that enhances the editability and animation control of 3D head avatars by incorporating 3D Gaussian Splatting (3DGS) as an explicit 3D representation.<n>Our approach delivers high-quality 3D-aware synthesis with state-of-the-art controllability.
  arXiv  Detail & Related papers  (2024-12-26T10:10:03Z)
• Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation [55.73399465968594]
  This paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss.
  arXiv  Detail & Related papers  (2024-04-02T11:03:24Z)
• Hybrid Explicit Representation for Ultra-Realistic Head Avatars [55.829497543262214]
  We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time.<n> UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures.<n>Experiments that our modeled results exceed those of state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-18T04:01:26Z)
• Designing a 3D-Aware StyleNeRF Encoder for Face Editing [15.303426697795143]
  We propose a 3D-aware encoder for GAN inversion and face editing based on the powerful StyleNeRF model. Our proposed 3Da encoder combines a parametric 3D face model with a learnable detail representation model to generate geometry, texture and view direction codes.
  arXiv  Detail & Related papers  (2023-02-19T03:32:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.