SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching

• URL: http://arxiv.org/abs/2608.06408v1
• Date: Mon, 03 Aug 2026 11:28:19 GMT
• Title: SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching
• Abstract summary: Audio-driven 3D facial animation aims to synthesize realistic and temporally coherent motions from speech.<n>We present SubtleTalk, a framework for generating natural and controllable weakly correlated facial dynamics.<n>Our method significantly improves the realism and diversity of weakly correlated facial dynamics while preserving accurate lip synchronization.
• Score: 21.891013914273714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-driven 3D facial animation aims to synthesize realistic and temporally coherent motions from speech. Despite notable progress in lip synchronization, weakly correlated dynamics, including eyebrow movements, eye blinks, and head motion, which are essential to photorealistic facial animation, remain difficult to model faithfully and often appear static or unnaturally repetitive. We attribute this limitation to three factors: (a) insufficient conditioning for weakly correlated dynamics; (b) the limited ability of deterministic regression to capture diverse motion patterns; (c) data bottlenecks from unreliable upper-face pseudo-labels and limited dataset diversity. To address these issues, we propose SubtleTalk, a framework for generating natural and controllable weakly correlated facial dynamics via multi-condition modeling and residual flow matching. First, to compensate for the limited guidance of speech alone, we introduce interpretable controls, including prosody, regional intensity, and Valence-Arousal signals, to explicitly capture the timing, magnitude, and affective variation of weakly correlated dynamics. Second, to overcome the limited expressiveness of deterministic regression, we build residual flow matching based on a stable speech-driven motion prior, allowing the model to capture stochastic deviations beyond deterministic prediction. Third, to alleviate the data bottleneck, we construct SubtleTalk-Face, a large-scale 3D facial animation dataset comprising about 3,900 identities and 74 hours of data, built via a simple and scalable pseudo-labeling pipeline and featuring improved upper-face tracking and frame-level VA annotations. Extensive experiments demonstrate that our method significantly improves the realism and diversity of weakly correlated facial dynamics while preserving accurate lip synchronization.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction [12.994613743334218]
  We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting.<n>To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure.<n>ECHO balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process.
  arXiv  Detail & Related papers  (2026-08-29T01:48:51Z)
• EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units [28.454399017725006]
  EmoZone-Talker is a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem.<n>Our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence.
  arXiv  Detail & Related papers  (2026-06-14T15:10:05Z)
• HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching [113.81911881001905]
  HO-Flow is a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects.<n>It first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold.<n>It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation.
  arXiv  Detail & Related papers  (2026-04-12T22:06:11Z)
• A Progressive Training Strategy for Vision-Language Models to Counteract Spatio-Temporal Hallucinations in Embodied Reasoning [49.61652671596548]
  "Multi-image hallucination reasoning", where a massive performance drop between forward and temporal queries reveals a dependence on superficial shortcuts instead of genuine understanding.<n>To mitigate this, we develop a new dataset: Chain-of-temporal building on detailed reasoning into Chain steps and definitive judgments.<n>Our experiments demonstrate that this approach not only improves accuracy but also the forward-backward performance gap from over 70% to only 6.53%.
  arXiv  Detail & Related papers  (2026-04-12T07:48:44Z)
• GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization [5.5806589487812746]
  GDPO-Listener is a novel framework that achieves highly expressive speaking and listening motion generation.<n>We introduce an Auto-Regressive Flow Matching architecture enabling stable supervised learning.<n>By isolating reward normalization across distinct FLAME parameter groups, GDPO explicitly incentivizes high variance expressive generations.
  arXiv  Detail & Related papers  (2026-03-26T04:36:27Z)
• NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control [59.6128550986024]
  NarraScore is a hierarchical framework predicated on the core insight that emotion serves as a high-density compression of narrative logic.<n>NarraScore employs a Dual-Branch Injection strategy to reconcile global structure with local dynamism.<n>NarraScore achieves state-of-the-art consistency and narrative alignment with negligible computational overhead.
  arXiv  Detail & Related papers  (2026-02-09T09:39:42Z)
• MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation [16.202732894319084]
  MoDiT is a novel framework that combines the 3D Morphable Model (3DMM) with a Diffusion-based Transformer.<n>Our contributions include: (i) A hierarchical denoising strategy with revised temporal attention and biased self/cross-attention mechanisms, enabling the model to refine lip synchronization.<n> (ii) The integration of 3DMM coefficients to provide explicit spatial constraints, ensuring accurate 3D-informed optical flow prediction.
  arXiv  Detail & Related papers  (2025-07-07T15:13:46Z)
• AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars [71.90109867684025]
  Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans.<n>We propose AsynFusion, a novel framework that leverages diffusion transformers to achieve cohesive expression and gesture synthesis.<n>AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations.
  arXiv  Detail & Related papers  (2025-05-21T03:28:53Z)
• TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting [21.474938045227702]
  We introduce TalkingGaussian, a deformation-based radiance fields framework for high-fidelity talking head synthesis. Our method renders high-quality lip-synchronized talking head videos, with better facial fidelity and higher efficiency compared with previous methods.
  arXiv  Detail & Related papers  (2024-04-23T17:55:07Z)
• Consistency Guided Scene Flow Estimation [159.24395181068218]
  CGSF is a self-supervised framework for the joint reconstruction of 3D scene structure and motion from stereo video. We show that the proposed model can reliably predict disparity and scene flow in challenging imagery. It achieves better generalization than the state-of-the-art, and adapts quickly and robustly to unseen domains.
  arXiv  Detail & Related papers  (2020-06-19T17:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.