AdaMesh: Personalized Facial Expressions and Head Poses for Adaptive Speech-Driven 3D Facial Animation

• URL: http://arxiv.org/abs/2310.07236v3
• Date: Thu, 20 Jun 2024 03:01:11 GMT
• Title: AdaMesh: Personalized Facial Expressions and Head Poses for Adaptive Speech-Driven 3D Facial Animation
• Authors: Liyang Chen, Weihong Bao, Shun Lei, Boshi Tang, Zhiyong Wu, Shiyin Kang, Haozhi Huang, Helen Meng,
• Abstract summary: AdaMesh is a novel adaptive speech-driven facial animation approach. It learns the personalized talking style from a reference video of about 10 seconds. It generates vivid facial expressions and head poses.
• Score: 49.4220768835379
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-driven 3D facial animation aims at generating facial movements that are synchronized with the driving speech, which has been widely explored recently. Existing works mostly neglect the person-specific talking style in generation, including facial expression and head pose styles. Several works intend to capture the personalities by fine-tuning modules. However, limited training data leads to the lack of vividness. In this work, we propose AdaMesh, a novel adaptive speech-driven facial animation approach, which learns the personalized talking style from a reference video of about 10 seconds and generates vivid facial expressions and head poses. Specifically, we propose mixture-of-low-rank adaptation (MoLoRA) to fine-tune the expression adapter, which efficiently captures the facial expression style. For the personalized pose style, we propose a pose adapter by building a discrete pose prior and retrieving the appropriate style embedding with a semantic-aware pose style matrix without fine-tuning. Extensive experimental results show that our approach outperforms state-of-the-art methods, preserves the talking style in the reference video, and generates vivid facial animation. The supplementary video and code will be available at https://adamesh.github.io.

Related papers

• MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes [74.82911268630463]
  Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. MimicTalk exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness.
  arXiv  Detail & Related papers  (2024-10-09T10:12:37Z)
• Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation [41.489700112318864]
  Speech-driven 3D facial animation aims to synthesize vivid facial animations that accurately synchronize with speech and match the unique speaking style. We introduce an innovative speaking style disentanglement method, which enables arbitrary-subject speaking style encoding. We also propose a novel framework called textbfMimic to learn disentangled representations of the speaking style and content from facial motions.
  arXiv  Detail & Related papers  (2023-12-18T01:49:42Z)
• Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape [19.431264557873117]
  We introduce VividTalker, a new framework designed to facilitate speech-driven 3D facial animation. We explicitly disentangle facial animation into head pose and mouth movement and encode them separately. We construct a new 3D dataset with detailed shapes and learn to synthesize facial details in line with speech content.
  arXiv  Detail & Related papers  (2023-10-31T07:47:19Z)
• Personalized Speech-driven Expressive 3D Facial Animation Synthesis with Style Control [1.8540152959438578]
  A realistic facial animation system should consider such identity-specific speaking styles and facial idiosyncrasies to achieve high-degree of naturalness and plausibility. We present a speech-driven expressive 3D facial animation synthesis framework that models identity specific facial motion as latent representations (called as styles) Our framework is trained in an end-to-end fashion and has a non-autoregressive encoder-decoder architecture with three main components.
  arXiv  Detail & Related papers  (2023-10-25T21:22:28Z)
• DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion [68.85904927374165]
  We propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. It captures the complex one-to-many relationships between speech and 3D face based on diffusion. It simultaneously achieves more realistic facial animation than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-23T04:14:55Z)
• VAST: Vivify Your Talking Avatar via Zero-Shot Expressive Facial Style Transfer [38.294607144065566]
  This paper proposes an unsupervised variational style transfer model (VAST) to vivify the neutral photo-realistic avatars. Our model consists of three key components: a style encoder that extracts facial style representations from the given video prompts; a hybrid facial expression decoder to model accurate speech-related movements. With our essential designs on facial style learning, our model is able to flexibly capture the expressive style from arbitrary video prompts and transfer it onto a personalized image in a zero-shot manner.
  arXiv  Detail & Related papers  (2023-08-09T09:38:14Z)
• Imitator: Personalized Speech-driven 3D Facial Animation [63.57811510502906]
  State-of-the-art methods deform the face topology of the target actor to sync the input audio without considering the identity-specific speaking style and facial idiosyncrasies of the target actor. We present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video. We show that our approach produces temporally coherent facial expressions from input audio while preserving the speaking style of the target actors.
  arXiv  Detail & Related papers  (2022-12-30T19:00:02Z)
• MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement [142.9900055577252]
  We propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. Our approach ensures highly accurate lip motion, while also plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion.
  arXiv  Detail & Related papers  (2021-04-16T17:05:40Z)
• Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose [67.31838207805573]
  We propose a deep neural network model that takes an audio signal A of a source person and a short video V of a target person as input. We outputs a synthesized high-quality talking face video with personalized head pose. Our method can generate high-quality talking face videos with more distinguishing head movement effects than state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-02-24T10:02:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.