AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

• URL: http://arxiv.org/abs/2605.07478v1
• Date: Fri, 08 May 2026 09:23:29 GMT
• Title: AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
• Abstract summary: Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion.<n>We propose AudioFace, a language-assisted framework for speech-driven blendshape generation.
• Score: 7.050132224945414
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.

Related papers

• TokTalk: Expressive Real-time Facial Animation from Audio-LLM Tokens [17.040246879872214]
  We present TokTalk, a system that directly outputs expressive facial animation in real-time from streaming audio-tokens.<n>A lightweight adaptation strategy allows our trained model to seamlessly connect to any token-based Audio-LLM at minimal computational overhead.<n>We show that the real-time performance of TokTalk is comparable in latency to prior art solutions, and significantly favorable in terms of quality, expressivity and control of the 3D facial performance.
  arXiv  Detail & Related papers  (2026-05-29T13:31:11Z)
• Polyglot: Multilingual Style Preserving Speech-Driven Facial Animation [24.33498946139279]
  Speech-Driven Facial Animation (SDFA) has gained significant attention due to its applications in movies, video games, and virtual reality.<n>In this work, we address multilingual SDFA, which is essential for realistic generation since language influences phonetics, rhythm, intonation, and facial expressions.<n>Existing methods typically rely on either language-specific or speaker-specific conditioning, but not both.<n>We introduce Polyglot, a unified diffusion-based architecture for personalized multilingual SDFA.
  arXiv  Detail & Related papers  (2026-04-17T14:43:33Z)
• A Bridge from Audio to Video: Phoneme-Viseme Alignment Allows Every Face to Speak Multiple Languages [60.81571443992153]
  Speech-driven talking face synthesis (TFS) focuses on generating facial animations from audio input.<n>Current models perform well in English but unsatisily in non-English languages, producing wrong mouth shapes and rigid facial expressions.<n>We propose Multilingual Experts (MuEx), a novel framework featuring a Phoneme-Guided Mixture-of-Experts architecture.
  arXiv  Detail & Related papers  (2025-10-08T03:46:39Z)
• Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering [53.2204901422631]
  Text2Lip is a viseme-centric framework that constructs an interpretable phonetic-visual bridge.<n>We show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness.
  arXiv  Detail & Related papers  (2025-08-04T12:50:22Z)
• VQTalker: Towards Multilingual Talking Avatars through Facial Motion Tokenization [20.728919218746363]
  VQTalker is a Vector Quantization-based framework for multilingual talking head generation.<n>Our approach is grounded in the phonetic principle that human speech comprises a finite set of distinct sound units.<n>VQTalker achieves state-of-the-art performance in both video-driven and speech-driven scenarios.
  arXiv  Detail & Related papers  (2024-12-13T06:14:57Z)
• GaussianSpeech: Audio-Driven Gaussian Avatars [76.10163891172192]
  We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio.<n>We propose a compact and efficient 3DGS-based avatar representation that generates expression-dependent color and leverages wrinkle- and perceptually-based losses to synthesize facial details.
  arXiv  Detail & Related papers  (2024-11-27T18:54:08Z)
• Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech [29.510756530126837]
  We introduce a data-driven method to visually represent articulator motion in MRI videos of the human vocal tract during speech. We leverage large pre-trained speech models, which are embedded with prior knowledge, to generalize the visual domain to unseen data.
  arXiv  Detail & Related papers  (2024-09-23T20:19:24Z)
• AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation [28.71632683090641]
  We propose an Audio-Visual Instruction system for expressive Talking face generation. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions.
  arXiv  Detail & Related papers  (2024-02-25T15:51:05Z)
• Imitator: Personalized Speech-driven 3D Facial Animation [63.57811510502906]
  State-of-the-art methods deform the face topology of the target actor to sync the input audio without considering the identity-specific speaking style and facial idiosyncrasies of the target actor. We present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video. We show that our approach produces temporally coherent facial expressions from input audio while preserving the speaking style of the target actors.
  arXiv  Detail & Related papers  (2022-12-30T19:00:02Z)
• Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis [67.73554826428762]
  We propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech, enabling efficient synthesis of clean, realistic speech from noisy signals.
  arXiv  Detail & Related papers  (2022-03-31T17:57:10Z)
• MakeItTalk: Speaker-Aware Talking-Head Animation [49.77977246535329]
  We present a method that generates expressive talking heads from a single facial image with audio as the only input. Based on this intermediate representation, our method is able to synthesize photorealistic videos of entire talking heads with full range of motion.
  arXiv  Detail & Related papers  (2020-04-27T17:56:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.