PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignment

• URL: http://arxiv.org/abs/2512.22602v1
• Date: Sat, 27 Dec 2025 14:14:14 GMT
• Title: PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignment
• Authors: Bin Wang, Yang Xu, Huan Zhao, Hao Zhang, Zixing Zhang,
• Abstract summary: "PTalker" is a novel framework for personalized 3D talking head animation.<n>It preserves speaking style through style disentanglement from audio and facial motion sequences.<n>It enhances lip-synchronization accuracy through a three-level alignment mechanism between audio and mesh modalities.
• Score: 19.347893658924367
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-driven 3D talking head generation aims to produce lifelike facial animations precisely synchronized with speech. While considerable progress has been made in achieving high lip-synchronization accuracy, existing methods largely overlook the intricate nuances of individual speaking styles, which limits personalization and realism. In this work, we present a novel framework for personalized 3D talking head animation, namely "PTalker". This framework preserves speaking style through style disentanglement from audio and facial motion sequences and enhances lip-synchronization accuracy through a three-level alignment mechanism between audio and mesh modalities. Specifically, to effectively disentangle style and content, we design disentanglement constraints that encode driven audio and motion sequences into distinct style and content spaces to enhance speaking style representation. To improve lip-synchronization accuracy, we adopt a modality alignment mechanism incorporating three aspects: spatial alignment using Graph Attention Networks to capture vertex connectivity in the 3D mesh structure, temporal alignment using cross-attention to capture and synchronize temporal dependencies, and feature alignment by top-k bidirectional contrastive losses and KL divergence constraints to ensure consistency between speech and mesh modalities. Extensive qualitative and quantitative experiments on public datasets demonstrate that PTalker effectively generates realistic, stylized 3D talking heads that accurately match identity-specific speaking styles, outperforming state-of-the-art methods. The source code and supplementary videos are available at: PTalker.

Related papers

• 3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars [46.32463788372058]
  3DXTalker is an expressive 3D talking avatar through data-curated identity modeling, audio-rich representations, and spatial dynamics controllability.<n>We introduce frame-wise amplitude and emotional cues beyond standard speech embeddings, ensuring superior lip synchronization and nuanced expression modulation.
  arXiv  Detail & Related papers  (2026-02-11T04:31:13Z)
• MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control [42.87874090062771]
  MirrorTalk is a generative framework based on a conditional diffusion model.<n>It can distill pure style representations from a brief reference video.<n>It achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.
  arXiv  Detail & Related papers  (2026-01-30T03:23:01Z)
• Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering [53.2204901422631]
  Text2Lip is a viseme-centric framework that constructs an interpretable phonetic-visual bridge.<n>We show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness.
  arXiv  Detail & Related papers  (2025-08-04T12:50:22Z)
• Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis [56.749927786910554]
  We propose a novel framework that integrates Gaussian Splatting with a structured Audio Factorization Plane (Audio-Plane) to enable high-quality, audio-synchronized, and real-time talking head generation.<n>Our method achieves state-of-the-art visual quality, precise audio-lip synchronization, and real-time performance, outperforming prior approaches across both 2D- and 3D-based paradigms.
  arXiv  Detail & Related papers  (2025-03-28T16:50:27Z)
• Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics [14.290468730787772]
  We introduce a speech-mesh synchronized representation that captures intricate correspondences between speech signals and 3D face meshes.<n>Experiments show that training 3D talking head generation models with our perceptual loss significantly improve all three aspects of perceptually accurate lip synchronization.
  arXiv  Detail & Related papers  (2025-03-26T08:18:57Z)
• GaussianSpeech: Audio-Driven Gaussian Avatars [76.10163891172192]
  We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio.<n>We propose a compact and efficient 3DGS-based avatar representation that generates expression-dependent color and leverages wrinkle- and perceptually-based losses to synthesize facial details.
  arXiv  Detail & Related papers  (2024-11-27T18:54:08Z)
• KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding [19.15471840100407]
  We present a novel approach for synthesizing 3D facial motions from audio sequences using key motion embeddings. Our method integrates linguistic and data-driven priors through two modules: the linguistic-based key motion acquisition and the cross-modal motion completion. The latter extends key motions into a full sequence of 3D talking faces guided by audio features, improving temporal coherence and audio-visual consistency.
  arXiv  Detail & Related papers  (2024-09-02T09:41:24Z)
• RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network [48.95833484103569]
  RealTalk is an audio-to-expression transformer and a high-fidelity expression-to-face framework. In the first component, we consider both identity and intra-personal variation features related to speaking lip movements. In the second component, we design a lightweight facial identity alignment (FIA) module. This novel design allows us to generate fine details in real-time, without depending on sophisticated and inefficient feature alignment modules.
  arXiv  Detail & Related papers  (2024-06-26T12:09:59Z)
• Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation [41.489700112318864]
  Speech-driven 3D facial animation aims to synthesize vivid facial animations that accurately synchronize with speech and match the unique speaking style. We introduce an innovative speaking style disentanglement method, which enables arbitrary-subject speaking style encoding. We also propose a novel framework called textbfMimic to learn disentangled representations of the speaking style and content from facial motions.
  arXiv  Detail & Related papers  (2023-12-18T01:49:42Z)
• DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion [68.85904927374165]
  We propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. It captures the complex one-to-many relationships between speech and 3D face based on diffusion. It simultaneously achieves more realistic facial animation than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-23T04:14:55Z)
• Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation [28.157431757281692]
  We propose a text-based talking-head video generation framework that synthesizes high-fidelity facial expressions and head motions. Our framework consists of a speaker-independent stage and a speaker-specific stage. Our algorithm achieves high-quality photo-realistic talking-head videos including various facial expressions and head motions according to speech rhythms.
  arXiv  Detail & Related papers  (2021-04-16T09:44:12Z)
• Learning Speech-driven 3D Conversational Gestures from Video [106.15628979352738]
  We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people.
  arXiv  Detail & Related papers  (2021-02-13T01:05:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.