EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis

• URL: http://arxiv.org/abs/2404.01647v1
• Date: Tue, 2 Apr 2024 05:32:39 GMT
• Title: EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis
• Abstract summary: This paper proposes a novel Efficient Disentanglement framework for Talking head generation (EDTalk) Our framework enables individual manipulation of mouth shape, head pose, and emotional expression, conditioned on video or audio inputs. Experiments are conducted to demonstrate the effectiveness of EDTalk.
• Score: 5.832760192773463
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving disentangled control over multiple facial motions and accommodating diverse input modalities greatly enhances the application and entertainment of the talking head generation. This necessitates a deep exploration of the decoupling space for facial features, ensuring that they a) operate independently without mutual interference and b) can be preserved to share with different modal input, both aspects often neglected in existing methods. To address this gap, this paper proposes a novel Efficient Disentanglement framework for Talking head generation (EDTalk). Our framework enables individual manipulation of mouth shape, head pose, and emotional expression, conditioned on video or audio inputs. Specifically, we employ three lightweight modules to decompose the facial dynamics into three distinct latent spaces representing mouth, pose, and expression, respectively. Each space is characterized by a set of learnable bases whose linear combinations define specific motions. To ensure independence and accelerate training, we enforce orthogonality among bases and devise an efficient training strategy to allocate motion responsibilities to each space without relying on external knowledge. The learned bases are then stored in corresponding banks, enabling shared visual priors with audio input. Furthermore, considering the properties of each space, we propose an Audio-to-Motion module for audio-driven talking head synthesis. Experiments are conducted to demonstrate the effectiveness of EDTalk. We recommend watching the project website: https://tanshuai0219.github.io/EDTalk/

Related papers

• Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions [84.73122243726775]
  Bagpiper is an 8B audio foundation model that interprets physical audio via rich captions.<n>During fine-tuning, Bagpiper adopts a caption-then-process workflow to solve diverse tasks without task-specific priors.<n>To the best of our knowledge, Bagpiper is among the first works that achieve unified understanding generation for general audio.
  arXiv  Detail & Related papers  (2026-02-05T02:20:07Z)
• EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis [3.939565545393492]
  This paper proposes EDTalk++, a novel full disentanglement framework for controllable talking head generation.<n>Our framework enables individual manipulation of mouth shape, head pose, eye movement, and emotional expression, conditioned on video or audio inputs.<n>Experiments are conducted to demonstrate the effectiveness of EDTalk++.
  arXiv  Detail & Related papers  (2025-08-19T01:55:25Z)
• SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion [78.77211425667542]
  SayAnything is a conditional video diffusion framework that directly synthesizes lip movements from audio input. Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation.
  arXiv  Detail & Related papers  (2025-02-17T07:29:36Z)
• Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion [6.677873152109559]
  Playmate is proposed to generate more lifelike facial expressions and talking faces. In the first stage, we introduce a decoupled implicit 3D representation to facilitate more accurate attribute disentanglement. In the second stage, we introduce an emotion-control module to encode emotion control information into the latent space.
  arXiv  Detail & Related papers  (2025-02-11T02:53:48Z)
• INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations [11.101103116878438]
  We propose INFP, a novel audio-driven head generation framework for dyadic interaction. INFP comprises a Motion-Based Head Imitation stage and an Audio-Guided Motion Generation stage. To facilitate this line of research, we introduce DyConv, a large scale dataset of rich dyadic conversations collected from the Internet.
  arXiv  Detail & Related papers  (2024-12-05T10:20:34Z)
• SPEAK: Speech-Driven Pose and Emotion-Adjustable Talking Head Generation [13.459396544300137]
  We propose a novel one-shot Talking Head Generation framework (SPEAK) that distinguishes itself from the general Talking Face Generation. We introduce Inter-Reconstructed Feature Disentanglement (IRFD) module to decouple facial features into three latent spaces. We then design a face editing module that modifies speech content and facial latent codes into a single latent space.
  arXiv  Detail & Related papers  (2024-05-12T11:41:44Z)
• FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio [45.71036380866305]
  We abstract the process of people hearing speech, extracting meaningful cues, and creating dynamically audio-consistent talking faces from a single audio. Specifically, it involves two critical challenges: one is to effectively decouple identity, content, and emotion from entangled audio, and the other is to maintain intra-video diversity and inter-video consistency. We introduce the Controllable Coherent Frame generation, which involves the flexible integration of three trainable adapters with frozen Latent Diffusion Models.
  arXiv  Detail & Related papers  (2024-03-04T09:59:48Z)
• Emotional Listener Portrait: Realistic Listener Motion Simulation in Conversation [50.35367785674921]
  Listener head generation centers on generating non-verbal behaviors of a listener in reference to the information delivered by a speaker. A significant challenge when generating such responses is the non-deterministic nature of fine-grained facial expressions during a conversation. We propose the Emotional Listener Portrait (ELP), which treats each fine-grained facial motion as a composition of several discrete motion-codewords. Our ELP model can not only automatically generate natural and diverse responses toward a given speaker via sampling from the learned distribution but also generate controllable responses with a predetermined attitude.
  arXiv  Detail & Related papers  (2023-09-29T18:18:32Z)
• Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos [69.79632907349489]
  We propose a self-supervised method for learning representations based on spatial audio-visual correspondences in egocentric videos. Our method uses a masked auto-encoding framework to synthesize masked (multi-channel) audio through the synergy of audio and vision.
  arXiv  Detail & Related papers  (2023-07-10T17:58:17Z)
• That's What I Said: Fully-Controllable Talking Face Generation [16.570649208028343]
  We propose a canonical space where every face has the same motion patterns but different identities. The second is to navigate a multimodal motion space that only represents motion-related features while eliminating identity information. Our method can generate natural-looking talking faces with fully controllable facial attributes and accurate lip synchronisation.
  arXiv  Detail & Related papers  (2023-04-06T17:56:50Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)
• Responsive Listening Head Generation: A Benchmark Dataset and Baseline [58.168958284290156]
  We define the responsive listening head generation task as the synthesis of a non-verbal head with motions and expressions reacting to the multiple inputs. Unlike speech-driven gesture or talking head generation, we introduce more modals in this task, hoping to benefit several research fields.
  arXiv  Detail & Related papers  (2021-12-27T07:18:50Z)
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation [96.66010515343106]
  We propose a clean yet effective framework to generate pose-controllable talking faces. We operate on raw face images, using only a single photo as an identity reference. Our model has multiple advanced capabilities including extreme view robustness and talking face frontalization.
  arXiv  Detail & Related papers  (2021-04-22T15:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.