PESTalk: Speech-Driven 3D Facial Animation with Personalized Emotional Styles

• URL: http://arxiv.org/abs/2512.05121v1
• Date: Mon, 13 Oct 2025 13:21:38 GMT
• Title: PESTalk: Speech-Driven 3D Facial Animation with Personalized Emotional Styles
• Abstract summary: PESTalk is a novel method for generating 3D facial animations with personalized emotional styles directly from speech.<n>It overcomes key limitations of existing approaches by introducing a Dual-Stream Emotion Extractor (DSEE) that captures both time and frequency-domain audio features.<n>It also introduces an Emotional Style Modeling Module (ESMM) that models individual expression patterns based on voiceprint characteristics.
• Score: 28.64103504776712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: PESTalk is a novel method for generating 3D facial animations with personalized emotional styles directly from speech. It overcomes key limitations of existing approaches by introducing a Dual-Stream Emotion Extractor (DSEE) that captures both time and frequency-domain audio features for fine-grained emotion analysis, and an Emotional Style Modeling Module (ESMM) that models individual expression patterns based on voiceprint characteristics. To address data scarcity, the method leverages a newly constructed 3D-EmoStyle dataset. Evaluations demonstrate that PESTalk outperforms state-of-the-art methods in producing realistic and personalized facial animations.

Related papers

• Human Feedback Driven Dynamic Speech Emotion Recognition [0.0]
  The study particularly focuses on the animation of emotional 3D avatars.<n>We propose a multi-stage method that includes the training of a classical speech emotion recognition model.<n>We introduce a novel approach to modeling emotional mixtures based on the Dirichlet distribution.
  arXiv  Detail & Related papers  (2025-08-18T17:25:27Z)
• MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization [8.422709223307722]
  Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style.<n>Previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference.<n>We propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input.
  arXiv  Detail & Related papers  (2025-07-28T06:47:59Z)
• MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding [48.54455964043634]
  MEDTalk is a novel framework for fine-grained and dynamic emotional talking head generation.<n>We integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions.<n>Our generated results can be conveniently integrated into the industrial production pipeline.
  arXiv  Detail & Related papers  (2025-07-08T15:14:27Z)
• EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models [66.67979602235015]
  EmoDiffusion is a novel approach that disentangles different emotions in speech to generate rich 3D emotional facial expressions.<n>We capture facial expressions under the guidance of animation experts using LiveLinkFace on an iPhone.
  arXiv  Detail & Related papers  (2025-03-14T02:54:22Z)
• MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes [74.82911268630463]
  Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. MimicTalk exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness.
  arXiv  Detail & Related papers  (2024-10-09T10:12:37Z)
• Personalized Speech-driven Expressive 3D Facial Animation Synthesis with Style Control [1.8540152959438578]
  A realistic facial animation system should consider such identity-specific speaking styles and facial idiosyncrasies to achieve high-degree of naturalness and plausibility. We present a speech-driven expressive 3D facial animation synthesis framework that models identity specific facial motion as latent representations (called as styles) Our framework is trained in an end-to-end fashion and has a non-autoregressive encoder-decoder architecture with three main components.
  arXiv  Detail & Related papers  (2023-10-25T21:22:28Z)
• DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion [68.85904927374165]
  We propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. It captures the complex one-to-many relationships between speech and 3D face based on diffusion. It simultaneously achieves more realistic facial animation than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-23T04:14:55Z)
• Emotional Speech-Driven Animation with Content-Emotion Disentanglement [51.34635009347183]
  We propose EMOTE, which generates 3D talking-head avatars that maintain lip-sync from speech while enabling explicit control over the expression of emotion. EmOTE produces speech-driven facial animations with better lip-sync than state-of-the-art methods trained on the same data.
  arXiv  Detail & Related papers  (2023-06-15T09:31:31Z)
• Imitator: Personalized Speech-driven 3D Facial Animation [63.57811510502906]
  State-of-the-art methods deform the face topology of the target actor to sync the input audio without considering the identity-specific speaking style and facial idiosyncrasies of the target actor. We present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video. We show that our approach produces temporally coherent facial expressions from input audio while preserving the speaking style of the target actors.
  arXiv  Detail & Related papers  (2022-12-30T19:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.