EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters

• URL: http://arxiv.org/abs/2503.19416v2
• Date: Wed, 02 Apr 2025 07:01:32 GMT
• Title: EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters
• Authors: Xuli Shen, Hua Cai, Dingding Yu, Weilin Shen, Qing Xu, Xiangyang Xue,
• Abstract summary: We present EmoHead to synthesize talking head videos via semantic expression parameters.<n>This work aims to enhance correlation from audio input across various emotions.<n>We leverage pre-trained hyperplane to refine facial movements by probing along the vertical direction.
• Score: 35.98628983591868
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generating emotion-specific talking head videos from audio input is an important and complex challenge for human-machine interaction. However, emotion is highly abstract concept with ambiguous boundaries, and it necessitates disentangled expression parameters to generate emotionally expressive talking head videos. In this work, we present EmoHead to synthesize talking head videos via semantic expression parameters. To predict expression parameter for arbitrary audio input, we apply an audio-expression module that can be specified by an emotion tag. This module aims to enhance correlation from audio input across various emotions. Furthermore, we leverage pre-trained hyperplane to refine facial movements by probing along the vertical direction. Finally, the refined expression parameters regularize neural radiance fields and facilitate the emotion-consistent generation of talking head videos. Experimental results demonstrate that semantic expression parameters lead to better reconstruction quality and controllability.

Related papers

• EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models [66.67979602235015]
  EmoDiffusion is a novel approach that disentangles different emotions in speech to generate rich 3D emotional facial expressions.<n>We capture facial expressions under the guidance of animation experts using LiveLinkFace on an iPhone.
  arXiv  Detail & Related papers  (2025-03-14T02:54:22Z)
• EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion [49.55774551366049]
  Diffusion models have revolutionized the field of talking head generation, yet still face challenges in expressiveness, controllability, and stability in long-time generation.<n>We propose an EmotiveTalk framework to address these issues.<n> Experimental results show that EmotiveTalk can generate expressive talking head videos, ensuring the promised controllability of emotions and stability during long-time generation.
  arXiv  Detail & Related papers  (2024-11-23T04:38:51Z)
• Audio-Driven Emotional 3D Talking-Head Generation [47.6666060652434]
  We present a novel system for synthesizing high-fidelity, audio-driven video portraits with accurate emotional expressions. We propose a pose sampling method that generates natural idle-state (non-speaking) videos in response to silent audio inputs.
  arXiv  Detail & Related papers  (2024-10-07T08:23:05Z)
• EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion [5.954758598327494]
  EMOdiffhead is a novel method for emotional talking head video generation. It enables fine-grained control of emotion categories and intensities. It achieves state-of-the-art performance compared to other emotion portrait animation methods.
  arXiv  Detail & Related papers  (2024-09-11T13:23:22Z)
• EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation [34.5592743467339]
  We propose a visual attribute-guided audio decoupler to generate fine-grained facial animations. To achieve more precise emotional expression, we introduce a fine-grained emotion coefficient prediction module. Our proposed method, EmoSpeaker, outperforms existing emotional talking face generation methods in terms of expression variation and lip synchronization.
  arXiv  Detail & Related papers  (2024-02-02T14:04:18Z)
• GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits [60.05683966405544]
  We present GMTalker, a Gaussian mixture-based emotional talking portraits generation framework. Specifically, we propose a continuous and disentangled latent space, achieving more flexible emotion manipulation. We also introduce a normalizing flow-based motion generator pretrained on a large dataset to generate diverse head poses, blinks, and eyeball movements.
  arXiv  Detail & Related papers  (2023-12-12T19:03:04Z)
• Emotional Speech-Driven Animation with Content-Emotion Disentanglement [51.34635009347183]
  We propose EMOTE, which generates 3D talking-head avatars that maintain lip-sync from speech while enabling explicit control over the expression of emotion. EmOTE produces speech-driven facial animations with better lip-sync than state-of-the-art methods trained on the same data.
  arXiv  Detail & Related papers  (2023-06-15T09:31:31Z)
• EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model [32.19539143308341]
  We propose the Emotion-Aware Motion Model (EAMM) to generate one-shot emotional talking faces. By incorporating the results from both modules, our method can generate satisfactory talking face results on arbitrary subjects.
  arXiv  Detail & Related papers  (2022-05-30T17:39:45Z)
• Speech Driven Talking Face Generation from a Single Image and an Emotion Condition [28.52180268019401]
  We propose a novel approach to rendering visual emotion expression in speech-driven talking face generation. We design an end-to-end talking face generation system that takes a speech utterance, a single face image, and a categorical emotion label as input. Objective evaluation on image quality, audiovisual synchronization, and visual emotion expression shows that the proposed system outperforms a state-of-the-art baseline system.
  arXiv  Detail & Related papers  (2020-08-08T20:46:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.