KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization

• URL: http://arxiv.org/abs/2606.28568v1
• Date: Fri, 26 Jun 2026 19:48:44 GMT
• Title: KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization
• Abstract summary: KM-Speaker is a novel keypoint-conditioned flow-based generative framework for speech-driven 3D facial animation.<n>It provides both global style guidance and frame-level temporal control from reference performances.<n> KM-Speaker consistently outperforms state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.
• Score: 8.476386736680002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-driven 3D facial animation methods face significant challenges in simultaneously achieving high-fidelity motion and precise artistic control at production quality. Existing controllable models typically learn global style control by relying on large-scale, low-quality \emph{in-the-wild} datasets that compromise overall animation realism. Furthermore, these frameworks often lack the fine-grained temporal precision required for demanding tasks such as dialogue localization (e.g., dubbing), where matching specific facial expressions is as critical as lip synchronization. We present KM-Speaker (Keypoint-Matching Speaker), a novel keypoint-conditioned flow-based generative framework that provides both global style guidance and frame-level temporal control from reference performances. We propose a disentanglement strategy that separates audio-driven lip motion from keypoint-driven upper-face dynamics, together with a global style context preservation mechanism to ensure coherent full-face expressiveness. KM-Speaker advances example-based 3D facial animation by achieving high-fidelity motion and flexible controllability in a data-constrained setting, consistently outperforming state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.

Related papers

• EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units [28.454399017725006]
  EmoZone-Talker is a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem.<n>Our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence.
  arXiv  Detail & Related papers  (2026-06-14T15:10:05Z)
• ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance [50.482989497576476]
  Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior.<n>Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion.<n>We propose ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model.
  arXiv  Detail & Related papers  (2026-06-11T13:00:58Z)
• UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars [25.55654497627044]
  UMo is a unified sparse motion modeling architecture for real-time co-speech avatars.<n>It efficiently performs real-time dense reconstruction, enabling temporally coherent and high-fidelity animation generation.<n>It preserves fine-grained speech-motion alignment even under strict latency constraints.
  arXiv  Detail & Related papers  (2026-05-14T11:56:03Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control [42.87874090062771]
  MirrorTalk is a generative framework based on a conditional diffusion model.<n>It can distill pure style representations from a brief reference video.<n>It achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.
  arXiv  Detail & Related papers  (2026-01-30T03:23:01Z)
• PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignment [19.347893658924367]
  "PTalker" is a novel framework for personalized 3D talking head animation.<n>It preserves speaking style through style disentanglement from audio and facial motion sequences.<n>It enhances lip-synchronization accuracy through a three-level alignment mechanism between audio and mesh modalities.
  arXiv  Detail & Related papers  (2025-12-27T14:14:14Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• Model See Model Do: Speech-Driven Facial Animation with Style Control [14.506128477193991]
  Speech-driven 3D facial animation plays a key role in applications such as virtual avatars, gaming, and digital content creation.<n>Existing methods have made significant progress in achieving accurate lip synchronization and generating basic emotional expressions.<n>We propose a novel example-based generation framework that conditions a latent diffusion model on a reference style clip.
  arXiv  Detail & Related papers  (2025-05-02T14:47:21Z)
• Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis [56.749927786910554]
  We propose a novel framework that integrates Gaussian Splatting with a structured Audio Factorization Plane (Audio-Plane) to enable high-quality, audio-synchronized, and real-time talking head generation.<n>Our method achieves state-of-the-art visual quality, precise audio-lip synchronization, and real-time performance, outperforming prior approaches across both 2D- and 3D-based paradigms.
  arXiv  Detail & Related papers  (2025-03-28T16:50:27Z)
• PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation [48.94486508604052]
  We introduce a novel, customizable one-shot audio-driven talking face generation framework, named PortraitTalk.<n>Our proposed method utilizes a latent diffusion framework consisting of two main components: IdentityNet and AnimateNet.<n>Key innovation of PortraitTalk is the incorporation of text prompts through decoupled cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-12-10T18:51:31Z)
• Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention [52.63080543011595]
  A novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. The proposed method can produce more realistic facial expressions and head posture movements.
  arXiv  Detail & Related papers  (2023-02-24T09:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.