Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation

• URL: http://arxiv.org/abs/2608.00663v1
• Date: Sat, 01 Aug 2026 13:33:38 GMT
• Title: Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation
• Abstract summary: GemTalk is a diffusion-based framework that combines the semantic richness of implicit representations with the structural precision of explicit geometric priors.<n>We introduce a Vision-guided Audio Emotion Projection (V-AEP) module to extract implicit emotional lip and expression features.<n>At the same time, a Diffusion-based Geometric Priors Generator (D-GPG) generates identity-aware blend coefficients as explicit structural priors.
• Score: 47.04292510603205
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Audio-driven emotional talking face generation aims to synthesize realistic videos with expressive facial dynamics. However, existing methods struggle to balance controllability and visual fidelity. Although implicit representations capture rich semantics, they lack structural guidance, often resulting in averaged emotional expressions. In contrast, explicit geometric methods offer better control over facial expressions but tend to sacrifice high-frequency texture details. To address it, we propose GemTalk, a diffusion-based framework that combines the semantic richness of implicit representations with the structural precision of explicit geometric priors. We introduce a Vision-guided Audio Emotion Projection (V-AEP) module to extract implicit emotional lip and expression features. At the same time, a Diffusion-based Geometric Priors Generator (D-GPG) generates identity-aware blendshape coefficients as explicit structural priors. Crucially, our Geometry-guided Emotion Modulation (GEM) module leverages these geometric priors to recalibrate the magnitude of implicit features, enabling precise, continuous control over emotional expressions, especially emotion intensity, without sacrificing visual quality. Extensive experiments show GemTalk achieves superior performance in photo-realism, and facial emotional dynamics.

Related papers

• SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization [14.632533340477591]
  SEDTalker is an emotion-aware framework for speech-driven 3D facial animation.<n>We use frame-level speech emotion diarization to achieve fine-grained expressive control.
  arXiv  Detail & Related papers  (2026-04-14T22:55:09Z)
• Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video [17.472242712450473]
  Cross-Modal Emotion Transfer (C-MET) is a novel approach that generates facial expressions based on speeches.<n>Our method improves emotion accuracy by 14% over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-09T04:28:03Z)
• EmoCAST: Emotional Talking Portrait via Emotive Text Description [56.42674612728354]
  EmoCAST is a diffusion-based framework for precise text-driven emotional synthesis.<n>In appearance modeling, emotional prompts are integrated through a text-guided decoupled emotive module.<n>EmoCAST achieves state-of-the-art performance in generating realistic, emotionally expressive, and audio-synchronized talking-head videos.
  arXiv  Detail & Related papers  (2025-08-28T10:02:06Z)
• Taming Transformer for Emotion-Controllable Talking Face Generation [61.835295250047196]
  We propose a novel method to tackle the emotion-controllable talking face generation task discretely.<n>Specifically, we employ two pre-training strategies to disentangle audio into independent components and quantize videos into combinations of visual tokens.<n>We conduct experiments on the MEAD dataset that controls the emotion of videos conditioned on multiple emotional audios.
  arXiv  Detail & Related papers  (2025-08-20T02:16:52Z)
• Think-Before-Draw: Decomposing Emotion Semantics & Fine-Grained Controllable Expressive Talking Head Generation [7.362433184546492]
  Emotional talking-head generation has emerged as a pivotal research area at the intersection of computer vision and multimodal artificial intelligence.<n>This study proposes the Think-Before-Draw framework to address two key challenges.
  arXiv  Detail & Related papers  (2025-07-17T03:33:46Z)
• MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding [48.54455964043634]
  MEDTalk is a novel framework for fine-grained and dynamic emotional talking head generation.<n>We integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions.<n>Our generated results can be conveniently integrated into the industrial production pipeline.
  arXiv  Detail & Related papers  (2025-07-08T15:14:27Z)
• Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation [63.94836524433559]
  DICE-Talk is a framework for disentangling identity with emotion and cooperating emotions with similar characteristics.<n>We develop a disentangled emotion embedder that jointly models audio-visual emotional cues through cross-modal attention.<n>Second, we introduce a correlation-enhanced emotion conditioning module with learnable Emotion Banks.<n>Third, we design an emotion discrimination objective that enforces affective consistency during the diffusion process.
  arXiv  Detail & Related papers  (2025-04-25T05:28:21Z)
• DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation [75.90730434449874]
  We introduce DREAM-Talk, a two-stage diffusion-based audio-driven framework, tailored for generating diverse expressions and accurate lip-sync concurrently. Given the strong correlation between lip motion and audio, we then refine the dynamics with enhanced lip-sync accuracy using audio features and emotion style. Both quantitatively and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of expressiveness, lip-sync accuracy and perceptual quality.
  arXiv  Detail & Related papers  (2023-12-21T05:03:18Z)
• High-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learning [43.09015109281053]
  We propose a more flexible and generalized framework for talking face generation. Specifically, we supplement the emotion style in text prompts and use an Aligned Multi-modal Emotion encoder to embed the text, image, and audio emotion modality into a unified space. An Emotion-aware Audio-to-3DMM Convertor is proposed to connect the emotion condition and the audio sequence to structural representation.
  arXiv  Detail & Related papers  (2023-05-04T05:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.