High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

• URL: http://arxiv.org/abs/2603.14781v1
• Date: Mon, 16 Mar 2026 03:25:10 GMT
• Title: High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions
• Abstract summary: We propose a novel approach by simultaneously refining both the latent code of a pretrained 3D-Aware GAN model for texture editing and the expression code of the driven 3DMM model for mesh editing.<n>To optimize the Dual Mappers, we propose a Text-Guided Optimization method, leveraging a CLIP-based objective function with expression text prompts as targets.
• Score: 83.73866089033585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Facial expression editing methods can be mainly categorized into two types based on their architectures: 2D-based and 3D-based methods. The former lacks 3D face modeling capabilities, making it difficult to edit 3D factors effectively. The latter has demonstrated superior performance in generating high-quality and view-consistent renderings using single-view 2D face images. Although these methods have successfully used animatable models to control facial expressions, they still have limitations in achieving precise control over fine-grained expressions. To address this issue, in this paper, we propose a novel approach by simultaneously refining both the latent code of a pretrained 3D-Aware GAN model for texture editing and the expression code of the driven 3DMM model for mesh editing. Specifically, we introduce a Dual Mappers module, comprising Texture Mapper and Emotion Mapper, to learn the transformations of the given latent code for textures and the expression code for meshes, respectively. To optimize the Dual Mappers, we propose a Text-Guided Optimization method, leveraging a CLIP-based objective function with expression text prompts as targets, while integrating a SubSpace Projection mechanism to project the text embedding to the expression subspace such that we can have more precise control over fine-grained expressions. Extensive experiments and comparative analyses demonstrate the effectiveness and superiority of our proposed method.

Related papers

• EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions [67.8321457457786]
  We introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations.<n>We present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model.<n>We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods.
  arXiv  Detail & Related papers  (2026-07-02T18:10:34Z)
• A Controllable 3D Deepfake Generation Framework with Gaussian Splatting [6.969908558294805]
  We propose a novel 3D deepfake generation framework based on 3D Gaussian Splatting.<n>It enables realistic, identity-preserving face swapping and reenactment in a fully controllable 3D space.<n>Our approach bridges the gap between 3D modeling and deepfake synthesis, enabling new directions for scene-aware, controllable, and immersive visual forgeries.
  arXiv  Detail & Related papers  (2025-09-15T06:34:17Z)
• Revealing Directions for Text-guided 3D Face Editing [52.85632020601518]
  3D face editing is a significant task in multimedia, aimed at the manipulation of 3D face models across various control signals. We present Face Clan, a text-general approach for generating and manipulating 3D faces based on arbitrary attribute descriptions. Our method offers a precisely controllable manipulation method, allowing users to intuitively customize regions of interest with the text description.
  arXiv  Detail & Related papers  (2024-10-07T12:04:39Z)
• Controllable 3D Face Generation with Conditional Style Code Diffusion [51.24656496304069]
  TEx-Face(TExt & Expression-to-Face) addresses challenges by dividing the task into three components, i.e., 3D GAN Inversion, Conditional Style Code Diffusion, and 3D Face Decoding. Experiments conducted on FFHQ, CelebA-HQ, and CelebA-Dialog demonstrate the promising performance of our TEx-Face.
  arXiv  Detail & Related papers  (2023-12-21T15:32:49Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields [52.14985242487535]
  We propose a new conditional 3D face synthesis framework, which enables 3D controllability over generated face images. At its core is a conditional Generative Occupancy Field (cGOF++) that effectively enforces the shape of the generated face to conform to a given 3D Morphable Model (3DMM) mesh. Experiments validate the effectiveness of the proposed method and show more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.
  arXiv  Detail & Related papers  (2022-11-23T19:02:50Z)
• Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars [36.4402388864691]
  3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images.
  arXiv  Detail & Related papers  (2022-11-21T06:40:46Z)
• IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis [38.517819699560945]
  Our system consists of three major components: (1) a 3D-semantics-aware generative model that produces view-consistent, disentangled face images and semantic masks; (2) a hybrid GAN inversion approach that initializes the latent codes from the semantic and texture encoder, and further optimized them for faithful reconstruction; and (3) a canonical editor that enables efficient manipulation of semantic masks in canonical view and product high-quality editing results.
  arXiv  Detail & Related papers  (2022-05-31T03:35:44Z)
• Convolutional Generation of Textured 3D Meshes [34.20939983046376]
  We propose a framework that can generate triangle meshes and associated high-resolution texture maps, using only 2D supervision from single-view natural images. A key contribution of our work is the encoding of the mesh and texture as 2D representations, which are semantically aligned and can be easily modeled by a 2D convolutional GAN. We demonstrate the efficacy of our method on Pascal3D+ Cars and CUB, both in an unconditional setting and in settings where the model is conditioned on class labels, attributes, and text.
  arXiv  Detail & Related papers  (2020-06-13T15:23:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.