Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control

• URL: http://arxiv.org/abs/2503.14517v1
• Date: Fri, 14 Mar 2025 02:52:41 GMT
• Title: Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control
• Abstract summary: 3D talking face method should offer both accurate lip synchronization and controllable expressions.<n>We propose a diffusion-transformer-based 3D talking face model, Cafe-Talk, which simultaneously incorporates coarse-temporal and fine-grained control conditions.<n>Experiments prove that Cafe-Talk achieves state-of-the-art lip synchronization and expressiveness and receives wide acceptance in fine-grained control in user studies.
• Score: 19.384766996505927
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech-driven 3D talking face method should offer both accurate lip synchronization and controllable expressions. Previous methods solely adopt discrete emotion labels to globally control expressions throughout sequences while limiting flexible fine-grained facial control within the spatiotemporal domain. We propose a diffusion-transformer-based 3D talking face generation model, Cafe-Talk, which simultaneously incorporates coarse- and fine-grained multimodal control conditions. Nevertheless, the entanglement of multiple conditions challenges achieving satisfying performance. To disentangle speech audio and fine-grained conditions, we employ a two-stage training pipeline. Specifically, Cafe-Talk is initially trained using only speech audio and coarse-grained conditions. Then, a proposed fine-grained control adapter gradually adds fine-grained instructions represented by action units (AUs), preventing unfavorable speech-lip synchronization. To disentangle coarse- and fine-grained conditions, we design a swap-label training mechanism, which enables the dominance of the fine-grained conditions. We also devise a mask-based CFG technique to regulate the occurrence and intensity of fine-grained control. In addition, a text-based detector is introduced with text-AU alignment to enable natural language user input and further support multimodal control. Extensive experimental results prove that Cafe-Talk achieves state-of-the-art lip synchronization and expressiveness performance and receives wide acceptance in fine-grained control in user studies. Project page: https://harryxd2018.github.io/cafe-talk/

Related papers

• EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units [28.454399017725006]
  EmoZone-Talker is a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem.<n>Our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence.
  arXiv  Detail & Related papers  (2026-06-14T15:10:05Z)
• MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation [45.88028371034407]
  MoCoTalk is a multi-conditional video diffusion framework that unifies four complementary control signals.<n> Adaptive Multi-Condition Router computes channel-wise, timestep-aware gating over the four condition streams.<n> Mouth-Augmented Shading Mesh is a 3DMM-based representation that decouples head motion, mouth motion, expression, and lighting.
  arXiv  Detail & Related papers  (2026-05-08T17:40:01Z)
• Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Router [72.29811385678168]
  We introduce Bind-Your-Avatar, an MM-DiT-based model specifically designed for multi-talking-character video generation in the same scene.<n>Specifically, we propose a novel framework incorporating a fine-grained Embedding Router that binds who' and speak what' together to address the audio-to-character correspondence control.
  arXiv  Detail & Related papers  (2025-06-24T17:50:16Z)
• MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis [56.25862714128288]
  This paper introduces textitMegaTTS 3, a zero-shot text-to-speech (TTS) system featuring an innovative sparse alignment algorithm. Specifically, we provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of alignment without limiting the search space. Experiments demonstrate that MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports highly flexible control over accent intensity.
  arXiv  Detail & Related papers  (2025-02-26T08:22:00Z)
• Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion [6.677873152109559]
  Playmate is proposed to generate more lifelike facial expressions and talking faces.<n>In the first stage, we introduce a decoupled implicit 3D representation to facilitate more accurate attribute disentanglement.<n>In the second stage, we introduce an emotion-control module to encode emotion control information into the latent space.
  arXiv  Detail & Related papers  (2025-02-11T02:53:48Z)
• XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation [72.12250272218792]
  We propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. We integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks. The generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings.
  arXiv  Detail & Related papers  (2024-11-20T12:02:12Z)
• Controllable Talking Face Generation by Implicit Facial Keypoints Editing [6.036277153327655]
  We present ControlTalk, a talking face generation method to control face expression deformation based on driven audio. Our experiments show that our method is superior to state-of-the-art performance on widely used benchmarks, including HDTF and MEAD.
  arXiv  Detail & Related papers  (2024-06-05T02:54:46Z)
• ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec [50.273832905535485]
  We present ControlSpeech, a text-to-speech (TTS) system capable of fully mimicking the speaker's voice and enabling arbitrary control and adjustment of speaking style. Prior zero-shot TTS models and controllable TTS models either could only mimic the speaker's voice without further control and adjustment capabilities or were unrelated to speaker-specific voice generation.
  arXiv  Detail & Related papers  (2024-06-03T11:15:16Z)
• EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis [5.832760192773463]
  This paper proposes a novel Efficient Disentanglement framework for Talking head generation (EDTalk) Our framework enables individual manipulation of mouth shape, head pose, and emotional expression, conditioned on video or audio inputs. Experiments are conducted to demonstrate the effectiveness of EDTalk.
  arXiv  Detail & Related papers  (2024-04-02T05:32:39Z)
• GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance [83.43852715997596]
  GSmoothFace is a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model. It can synthesize smooth lip dynamics while preserving the speaker's identity. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality.
  arXiv  Detail & Related papers  (2023-12-12T16:00:55Z)
• FaceFormer: Speech-Driven 3D Facial Animation with Transformers [46.8780140220063]
  Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data. We propose a Transformer-based autoregressive model, FaceFormer, which encodes the long-term audio context and autoregressively predicts a sequence of animated 3D face meshes.
  arXiv  Detail & Related papers  (2021-12-10T04:21:59Z)
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation [96.66010515343106]
  We propose a clean yet effective framework to generate pose-controllable talking faces. We operate on raw face images, using only a single photo as an identity reference. Our model has multiple advanced capabilities including extreme view robustness and talking face frontalization.
  arXiv  Detail & Related papers  (2021-04-22T15:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.