TDMM-LM: Bridging Facial Understanding and Animation via Language Models

• URL: http://arxiv.org/abs/2603.16936v1
• Date: Sat, 14 Mar 2026 15:57:53 GMT
• Title: TDMM-LM: Bridging Facial Understanding and Animation via Language Models
• Abstract summary: We leverage foundation generative models to synthesize a large, balanced corpus of facial behavior.<n>We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters.<n>Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks.
• Score: 46.82447117476669
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-guided human body animation has advanced rapidly, yet facial animation lags due to the scarcity of well-annotated, text-paired facial corpora. To close this gap, we leverage foundation generative models to synthesize a large, balanced corpus of facial behavior. We design prompts suite covering emotions and head motions, generate about 80 hours of facial videos with multiple generators, and fit per-frame 3D facial parameters, yielding large-scale (prompt and parameter) pairs for training. Building on this dataset, we probe language models for bidirectional competence over facial motion via two complementary tasks: (1) Motion2Language: given a sequence of 3D facial parameters, the model produces natural-language descriptions capturing content, style, and dynamics; and (2) Language2Motion: given a prompt, the model synthesizes the corresponding sequence of 3D facial parameters via quantized motion tokens for downstream animation. Extensive experiments show that in this setting language models can both interpret and synthesize facial motion with strong generalization. To best of our knowledge, this is the first work to cast facial-parameter modeling as a language problem, establishing a unified path for text-conditioned facial animation and motion understanding.

Related papers

• GaussianSpeech: Audio-Driven Gaussian Avatars [76.10163891172192]
  We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio.<n>We propose a compact and efficient 3DGS-based avatar representation that generates expression-dependent color and leverages wrinkle- and perceptually-based losses to synthesize facial details.
  arXiv  Detail & Related papers  (2024-11-27T18:54:08Z)
• JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation [10.003794924759765]
  JoyVASA is a diffusion-based method for generating facial dynamics and head motion in audio-driven facial animation.<n>We introduce a decoupled facial representation framework that separates dynamic facial expressions from static 3D facial representations.<n>In the second stage, a diffusion transformer is trained to generate motion sequences directly from audio cues, independent of character identity.
  arXiv  Detail & Related papers  (2024-11-14T06:13:05Z)
• MMHead: Towards Fine-grained Multi-modal 3D Facial Animation [68.04052669266174]
  We construct a large-scale multi-modal 3D facial animation dataset, MMHead. MMHead consists of 49 hours of 3D facial motion sequences, speech audios, and rich hierarchical text annotations. Based on the MMHead dataset, we establish benchmarks for two new tasks: text-induced 3D talking head animation and text-to-3D facial motion generation.
  arXiv  Detail & Related papers  (2024-10-10T09:37:01Z)
• Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance [41.692420421029695]
  We introduce an efficient variational auto-encoder mapping facial geometry and images to a highly generalized expression latent space. We then use GNPFA to extract high-quality expressions and accurate head poses from a large array of videos. We propose Media2Face, a diffusion model in GNPFA latent space for co-speech facial animation generation.
  arXiv  Detail & Related papers  (2024-01-28T16:17:59Z)
• Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation [41.489700112318864]
  Speech-driven 3D facial animation aims to synthesize vivid facial animations that accurately synchronize with speech and match the unique speaking style. We introduce an innovative speaking style disentanglement method, which enables arbitrary-subject speaking style encoding. We also propose a novel framework called textbfMimic to learn disentangled representations of the speaking style and content from facial motions.
  arXiv  Detail & Related papers  (2023-12-18T01:49:42Z)
• 3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing [22.30870274645442]
  We present 3DiFACE, a novel method for personalized speech-driven 3D facial animation and editing. Our method outperforms existing state-of-the-art techniques and yields speech-driven animations with greater fidelity and diversity.
  arXiv  Detail & Related papers  (2023-12-01T19:01:05Z)
• Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape [19.431264557873117]
  We introduce VividTalker, a new framework designed to facilitate speech-driven 3D facial animation. We explicitly disentangle facial animation into head pose and mouth movement and encode them separately. We construct a new 3D dataset with detailed shapes and learn to synthesize facial details in line with speech content.
  arXiv  Detail & Related papers  (2023-10-31T07:47:19Z)
• DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion [68.85904927374165]
  We propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. It captures the complex one-to-many relationships between speech and 3D face based on diffusion. It simultaneously achieves more realistic facial animation than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-23T04:14:55Z)
• Language-Guided Face Animation by Recurrent StyleGAN-based Generator [87.56260982475564]
  We study a novel task, language-guided face animation, that aims to animate a static face image with the help of languages. We propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames.
  arXiv  Detail & Related papers  (2022-08-11T02:57:30Z)
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement [142.9900055577252]
  We propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. Our approach ensures highly accurate lip motion, while also plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion.
  arXiv  Detail & Related papers  (2021-04-16T17:05:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.