MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control

• URL: http://arxiv.org/abs/2601.22501v1
• Date: Fri, 30 Jan 2026 03:23:01 GMT
• Title: MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control
• Authors: Renjie Lu, Xulong Zhang, Xiaoyang Qu, Jianzong Wang, Shangfei Wang,
• Abstract summary: MirrorTalk is a generative framework based on a conditional diffusion model.<n>It can distill pure style representations from a brief reference video.<n>It achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.
• Score: 42.87874090062771
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthesizing personalized talking faces that uphold and highlight a speaker's unique style while maintaining lip-sync accuracy remains a significant challenge. A primary limitation of existing approaches is the intrinsic confounding of speaker-specific talking style and semantic content within facial motions, which prevents the faithful transfer of a speaker's unique persona to arbitrary speech. In this paper, we propose MirrorTalk, a generative framework based on a conditional diffusion model, combined with a Semantically-Disentangled Style Encoder (SDSE) that can distill pure style representations from a brief reference video. To effectively utilize this representation, we further introduce a hierarchical modulation strategy within the diffusion process. This mechanism guides the synthesis by dynamically balancing the contributions of audio and style features across distinct facial regions, ensuring both precise lip-sync accuracy and expressive full-face dynamics. Extensive experiments demonstrate that MirrorTalk achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.

Related papers

• PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignment [19.347893658924367]
  "PTalker" is a novel framework for personalized 3D talking head animation.<n>It preserves speaking style through style disentanglement from audio and facial motion sequences.<n>It enhances lip-synchronization accuracy through a three-level alignment mechanism between audio and mesh modalities.
  arXiv  Detail & Related papers  (2025-12-27T14:14:14Z)
• HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis [90.74616208952791]
  HM-Talker is a novel framework for generating high-fidelity, temporally coherent talking heads.<n>Explicit cues use Action Units (AUs), anatomically defined facial muscle movements, alongside implicit features to minimize phoneme-viseme misalignment.
  arXiv  Detail & Related papers  (2025-08-14T12:01:52Z)
• Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering [53.2204901422631]
  Text2Lip is a viseme-centric framework that constructs an interpretable phonetic-visual bridge.<n>We show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness.
  arXiv  Detail & Related papers  (2025-08-04T12:50:22Z)
• Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversion [53.26424100244925]
  Expressive voice conversion aims to transfer both speaker identity and expressive attributes from a target speech to a given source speech.<n>In this work, we improve over a self-supervised, non-autoregressive framework with a conditional variational autoencoder.
  arXiv  Detail & Related papers  (2025-06-04T14:42:12Z)
• OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers [18.187498205054748]
  We present OmniSync, a universal lip synchronization framework for diverse visual scenarios.<n>Our approach introduces a mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks.<n>We also establish the AIGCLipSync Benchmark, the first evaluation suite for lip sync in AI-generated videos.
  arXiv  Detail & Related papers  (2025-05-27T17:20:38Z)
• Model See Model Do: Speech-Driven Facial Animation with Style Control [14.506128477193991]
  Speech-driven 3D facial animation plays a key role in applications such as virtual avatars, gaming, and digital content creation.<n>Existing methods have made significant progress in achieving accurate lip synchronization and generating basic emotional expressions.<n>We propose a novel example-based generation framework that conditions a latent diffusion model on a reference style clip.
  arXiv  Detail & Related papers  (2025-05-02T14:47:21Z)
• Shushing! Let's Imagine an Authentic Speech from the Silent Video [15.426152742881365]
  Vision-guided speech generation aims to produce authentic speech from facial appearance or lip motions without relying on auditory signals.<n>Despite recent progress, existing methods struggle to achieve unified cross-modal alignment across semantics, timbre, and emotional prosody from visual cues.<n>We introduce ImaginTalk, a novel cross-modal diffusion framework that generates faithful speech using only visual input.
  arXiv  Detail & Related papers  (2025-03-19T06:28:17Z)
• High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model [89.29655924125461]
  We propose a novel landmark-based diffusion model for talking face generation. We first establish the less ambiguous mapping from audio to landmark motion of lip and jaw. Then, we introduce an innovative conditioning module called TalkFormer to align the synthesized motion with the motion represented by landmarks.
  arXiv  Detail & Related papers  (2024-08-10T02:58:28Z)
• Style-Preserving Lip Sync via Audio-Aware Style Reference [85.686411962494]
  Individuals exhibit distinct lip shapes when speaking the same utterance, attributed to the unique speaking styles of individuals.<n>We develop an advanced Transformer-based model adept at predicting lip motion corresponding to the input audio, augmented by the style information aggregated through cross-attention layers from style reference video.<n>Experiments validate the efficacy of the proposed approach in achieving precise lip sync, preserving speaking styles, and generating high-fidelity, realistic talking face videos.
  arXiv  Detail & Related papers  (2024-08-10T02:46:11Z)
• RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network [48.95833484103569]
  RealTalk is an audio-to-expression transformer and a high-fidelity expression-to-face framework. In the first component, we consider both identity and intra-personal variation features related to speaking lip movements. In the second component, we design a lightweight facial identity alignment (FIA) module. This novel design allows us to generate fine details in real-time, without depending on sophisticated and inefficient feature alignment modules.
  arXiv  Detail & Related papers  (2024-06-26T12:09:59Z)
• Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation [29.87407471246318]
  This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations. Our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities.
  arXiv  Detail & Related papers  (2024-06-13T04:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.