Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillation

• URL: http://arxiv.org/abs/2507.18352v1
• Date: Thu, 24 Jul 2025 12:25:12 GMT
• Title: Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillation
• Abstract summary: We explore on-device, real-time facial animation models in the context of game development.<n>We employ a high-performing teacher model to train very small student models.<n>We can reduce the memory footprint to up to 3.4 MB and required future audio context to up to 81 ms.
• Score: 11.486241310290954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The training of high-quality, robust machine learning models for speech-driven 3D facial animation requires a large, diverse dataset of high-quality audio-animation pairs. To overcome the lack of such a dataset, recent work has introduced large pre-trained speech encoders that are robust to variations in the input audio and, therefore, enable the facial animation model to generalize across speakers, audio quality, and languages. However, the resulting facial animation models are prohibitively large and lend themselves only to offline inference on a dedicated machine. In this work, we explore on-device, real-time facial animation models in the context of game development. We overcome the lack of large datasets by using hybrid knowledge distillation with pseudo-labeling. Given a large audio dataset, we employ a high-performing teacher model to train very small student models. In contrast to the pre-trained speech encoders, our student models only consist of convolutional and fully-connected layers, removing the need for attention context or recurrent updates. In our experiments, we demonstrate that we can reduce the memory footprint to up to 3.4 MB and required future audio context to up to 81 ms while maintaining high-quality animations. This paves the way for on-device inference, an important step towards realistic, model-driven digital characters.

Related papers

• AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model [14.363854681138669]
  We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data.<n>We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning technique.<n>We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters.
  arXiv  Detail & Related papers  (2026-08-17T05:53:42Z)
• FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences [64.71722244060001]
  We present a framework that learns an automatic perceptual model from human preference data to improve audio-driven 3D facial animation.<n>We propose a Facial Motion Reward model, termed FMReward, which takes audio and 3D facial motion as inputs and predicts a perceptual quality score aligned with human preferences.
  arXiv  Detail & Related papers  (2026-08-15T15:54:34Z)
• StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model [73.30619724574642]
  Speech-driven 3D facial animation aims to generate realistic and synchronized facial motions driven by speech inputs.<n>Recent methods have employed audio-conditioned diffusion models for 3D facial animation.<n>We propose a novel autoregressive diffusion model that processes audio in a streaming manner.
  arXiv  Detail & Related papers  (2025-11-18T07:55:16Z)
• EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation [8.214084596349744]
  We propose a novel unified Multi-Task paradigm for human animation, treating diverse generation tasks as spatial-temporal local reconstructions.<n>We introduce a multi-modal decoupled cross-attention module to fuse multi-modals in a divide-and-conquer manner.<n>We propose a new SFT+Reward alternating training paradigm, enabling the minimal model with 1.3B parameters to achieve generation quality comparable to models with 10 times the parameters count.
  arXiv  Detail & Related papers  (2025-07-05T05:36:26Z)
• TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models [2.176487921193175]
  TalkingMachines is an efficient framework that transforms pretrained video generation models into real-time, audio-driven character animators.<n>TalkingMachines enables natural conversational experiences by integrating an audio large language model (LLM) with our video generation foundation model.
  arXiv  Detail & Related papers  (2025-06-03T17:29:28Z)
• Vision-Speech Models: Teaching Speech Models to Converse about Images [67.62394024470528]
  We introduce MoshiVis, augmenting a recent dialogue speech LLM, Moshi, with visual inputs through lightweight adaptation modules.<n>An additional dynamic gating mechanism enables the model to more easily switch between the visual inputs and unrelated conversation topics.<n>We evaluate the model on downstream visual understanding tasks with both audio and text prompts, and report qualitative samples of interactions with MoshiVis.
  arXiv  Detail & Related papers  (2025-03-19T18:40:45Z)
• MagicArticulate: Make Your 3D Models Articulation-Ready [109.35703811628045]
  We present MagicArticulate, an effective framework that automatically transforms static 3D models into articulation-ready assets.<n>Our key contributions are threefold. First, we introduce Articulation-averse benchmark containing over 33k 3D models with high-quality articulation annotations, carefully curated from XL-XL.<n>Extensive experiments demonstrate that MagicArticulate significantly outperforms existing methods across diverse object categories.
  arXiv  Detail & Related papers  (2025-02-17T18:53:27Z)
• UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model [4.443066817473078]
  We present UniTalker, a unified model featuring a multi-head architecture designed to leverage datasets with varied annotations. To enhance training stability and ensure consistency among multi-head outputs, we employ three training strategies, namely, PCA, model warm-up, and pivot identity embedding. With a single trained UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for BIWI dataset and 13.7% for Vocaset.
  arXiv  Detail & Related papers  (2024-08-01T17:59:27Z)
• Pheme: Efficient and Conversational Speech Generation [52.34331755341856]
  We introduce the Pheme model series that offers compact yet high-performing conversational TTS models. It can be trained efficiently on smaller-scale conversational data, cutting data demands by more than 10x but still matching the quality of the autoregressive TTS models.
  arXiv  Detail & Related papers  (2024-01-05T14:47:20Z)
• Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications [20.842799581850617]
  We consider the task of animating 3D facial geometry from speech signal. Existing works are primarily deterministic, focusing on learning a one-to-one mapping from speech signal to 3D face meshes on small datasets with limited speakers.
  arXiv  Detail & Related papers  (2023-11-30T01:14:43Z)
• DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation [10.73030153404956]
  We propose a cross-modal dual-learning framework, termed DualTalker, to improve data usage efficiency. The framework is trained jointly with the primary task (audio-driven facial animation) and its dual task (lip reading) and shares common audio/motion encoder components. Our approach outperforms current state-of-the-art methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2023-11-08T15:39:56Z)
• Probabilistic Adaptation of Text-to-Video Models [181.84311524681536]
  Video Adapter is capable of incorporating the broad knowledge and preserving the high fidelity of a large pretrained video model in a task-specific small video model. Video Adapter is able to generate high-quality yet specialized videos on a variety of tasks such as animation, egocentric modeling, and modeling of simulated and real-world robotics data.
  arXiv  Detail & Related papers  (2023-06-02T19:00:17Z)
• Generating Holistic 3D Human Motion from Speech [97.11392166257791]
  We build a high-quality dataset of 3D holistic body meshes with synchronous speech. We then define a novel speech-to-motion generation framework in which the face, body, and hands are modeled separately.
  arXiv  Detail & Related papers  (2022-12-08T17:25:19Z)
• AnimeCeleb: Large-Scale Animation CelebFaces Dataset via Controllable 3D Synthetic Models [19.6347170450874]
  We present a large-scale animation celebfaces dataset (AnimeCeleb) via controllable synthetic animation models. To facilitate the data generation process, we build a semi-automatic pipeline based on an open 3D software. This leads to constructing a large-scale animation face dataset that includes multi-pose and multi-style animation faces with rich annotations.
  arXiv  Detail & Related papers  (2021-11-15T10:00:06Z)
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement [142.9900055577252]
  We propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. Our approach ensures highly accurate lip motion, while also plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion.
  arXiv  Detail & Related papers  (2021-04-16T17:05:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.