EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units

• URL: http://arxiv.org/abs/2606.15848v1
• Date: Sun, 14 Jun 2026 15:10:05 GMT
• Title: EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units
• Abstract summary: EmoZone-Talker is a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem.<n>Our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence.
• Score: 28.454399017725006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Gaussian Splatting (3DGS) has shown strong potential for high-fidelity talking head synthesis. However, enabling fine-grained, interpretable, and editable facial expression control remains fundamentally challenging due to intrinsic conflicts between speech-driven facial dynamics and explicit expression signals. Existing methods rely on implicit multimodal fusion, leading to spatial entanglement and temporal instability. We present EmoZone-Talker, a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem under cross-modal conflicts. Our approach introduces an explicit spatial disentanglement and temporal dynamics modeling of facial motion. Specifically, we propose Synergy Zones with Prioritized Attention Bias (SZ-PAB) to explicitly decouple modality contributions via region-wise constraints guided by anatomical priors, and a Channel-Independent Temporal AU Encoder (CIT-AE) to model temporally coherent AU dynamics. By integrating these representations into 3D Gaussian deformation, EmoZone-Talker enables precise and interpretable control over facial expressions. Extensive experiments demonstrate that our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence, while preserving high rendering quality and accurate lip synchronization. Code will be publicly released to facilitate reproducibility and further research.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction [12.994613743334218]
  We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting.<n>To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure.<n>ECHO balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process.
  arXiv  Detail & Related papers  (2026-08-29T01:48:51Z)
• KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization [8.476386736680002]
  KM-Speaker is a novel keypoint-conditioned flow-based generative framework for speech-driven 3D facial animation.<n>It provides both global style guidance and frame-level temporal control from reference performances.<n> KM-Speaker consistently outperforms state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.
  arXiv  Detail & Related papers  (2026-06-26T19:48:44Z)
• Talking Head Generation via AU-Guided Landmark Prediction [48.30051606459973]
  We propose a two-stage framework for audio-driven talking head generation with fine-grained expression control via facial Action Units (AUs)<n>In the first stage, a variational motion generator predicts temporally coherent landmark sequences from audio and AU intensities.<n>In the second stage, a diffusion-based synthesizer generates realistic, lip-synced videos conditioned on these landmarks and a reference image.
  arXiv  Detail & Related papers  (2025-09-24T04:01:57Z)
• HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis [90.74616208952791]
  HM-Talker is a novel framework for generating high-fidelity, temporally coherent talking heads.<n>Explicit cues use Action Units (AUs), anatomically defined facial muscle movements, alongside implicit features to minimize phoneme-viseme misalignment.
  arXiv  Detail & Related papers  (2025-08-14T12:01:52Z)
• Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis [56.749927786910554]
  We propose a novel framework that integrates Gaussian Splatting with a structured Audio Factorization Plane (Audio-Plane) to enable high-quality, audio-synchronized, and real-time talking head generation.<n>Our method achieves state-of-the-art visual quality, precise audio-lip synchronization, and real-time performance, outperforming prior approaches across both 2D- and 3D-based paradigms.
  arXiv  Detail & Related papers  (2025-03-28T16:50:27Z)
• DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model [7.165879904419689]
  DisentTalk presents a data-driven semantic disentanglement framework that decomposes 3DMM expression parameters into meaningful subspaces for fine-grained facial control.<n>To address the scarcity of high-quality Chinese training data, we introduce CHDTF, a Chinese high-definition talking face dataset.
  arXiv  Detail & Related papers  (2025-03-24T11:46:34Z)
• TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting [21.474938045227702]
  We introduce TalkingGaussian, a deformation-based radiance fields framework for high-fidelity talking head synthesis. Our method renders high-quality lip-synchronized talking head videos, with better facial fidelity and higher efficiency compared with previous methods.
  arXiv  Detail & Related papers  (2024-04-23T17:55:07Z)
• FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models [85.16273912625022]
  We introduce FaceTalk, a novel generative approach designed for synthesizing high-fidelity 3D motion sequences of talking human heads from audio signal. To the best of our knowledge, this is the first work to propose a generative approach for realistic and high-quality motion synthesis of human heads.
  arXiv  Detail & Related papers  (2023-12-13T19:01:07Z)
• Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis [20.111316792226482]
  ER-NeRF is a conditional Neural Radiance Fields (NeRF) based architecture for talking portrait. Our method renders better high-fidelity and audio-lips, with realistic details and high efficiency compared to previous methods.
  arXiv  Detail & Related papers  (2023-07-18T15:07:39Z)
• Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention [52.63080543011595]
  A novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. The proposed method can produce more realistic facial expressions and head posture movements.
  arXiv  Detail & Related papers  (2023-02-24T09:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.