LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild

• URL: http://arxiv.org/abs/2602.00189v1
• Date: Fri, 30 Jan 2026 08:02:49 GMT
• Title: LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild
• Authors: Zhipeng Chen, Xinheng Wang, Lun Xie, Haijie Yuan, Hang Pan,
• Abstract summary: This paper proposes a generic method, LPIPS-AttnWav2Lip, for reconstructing face images of any speaker based on audio.<n>The proposed method achieves outstanding performance in terms of lip synchronization accuracy and visual quality.
• Score: 9.682333912273906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Researchers have shown a growing interest in Audio-driven Talking Head Generation. The primary challenge in talking head generation is achieving audio-visual coherence between the lips and the audio, known as lip synchronization. This paper proposes a generic method, LPIPS-AttnWav2Lip, for reconstructing face images of any speaker based on audio. We used the U-Net architecture based on residual CBAM to better encode and fuse audio and visual modal information. Additionally, the semantic alignment module extends the receptive field of the generator network to obtain the spatial and channel information of the visual features efficiently; and match statistical information of visual features with audio latent vector to achieve the adjustment and injection of the audio content information to the visual information. To achieve exact lip synchronization and to generate realistic high-quality images, our approach adopts LPIPS Loss, which simulates human judgment of image quality and reduces instability possibility during the training process. The proposed method achieves outstanding performance in terms of lip synchronization accuracy and visual quality as demonstrated by subjective and objective evaluation results. The code for the paper is available at the following link: https://github.com/FelixChan9527/LPIPS-AttnWav2Lip

Related papers

• Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework [56.30142869506262]
  Inpainting-based talking face generation aims to preserve video details such as pose, lighting, and gestures while modifying only lip motion.<n>This mechanism can introduce lip leaking, where generated lips are influenced by the reference image rather than solely by the driving audio.<n>We propose a systematic evaluation methodology to analyze and quantify lip leakage.
  arXiv  Detail & Related papers  (2025-11-05T17:11:53Z)
• Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering [53.2204901422631]
  Text2Lip is a viseme-centric framework that constructs an interpretable phonetic-visual bridge.<n>We show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness.
  arXiv  Detail & Related papers  (2025-08-04T12:50:22Z)
• Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation [51.92522679353731]
  We propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. We introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance.
  arXiv  Detail & Related papers  (2024-05-07T13:55:50Z)
• Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization [38.64540967776744]
  Diff2Lip is an audio-conditioned diffusion-based model which is able to do lip synchronization in-the-wild while preserving these qualities. We show results on both reconstruction (same audio-video inputs) as well as cross (different audio-video inputs) settings on Voxceleb2 and LRW datasets.
  arXiv  Detail & Related papers  (2023-08-18T17:59:40Z)
• Audio-driven Talking Face Generation with Stabilized Synchronization Loss [60.01529422759644]
  Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality. We first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization.
  arXiv  Detail & Related papers  (2023-07-18T15:50:04Z)
• Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert [89.07178484337865]
  Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. Previous studies revealed the importance of lip-speech synchronization and visual quality. We propose using a lip-reading expert to improve the intelligibility of the generated lip regions.
  arXiv  Detail & Related papers  (2023-03-29T07:51:07Z)
• Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers [91.00397473678088]
  Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. We propose the Audio-Visual Context-Aware Transformer (AV-CAT) framework, which produces accurate lip-sync with photo-realistic quality. Our model can generate high-fidelity lip-synced results for arbitrary subjects.
  arXiv  Detail & Related papers  (2022-12-09T16:32:46Z)
• SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory [27.255990661166614]
  The challenge of talking face generation from speech lies in aligning two different modal information, audio and video, such that the mouth region corresponds to input audio. Previous methods either exploit audio-visual representation learning or leverage intermediate structural information such as landmarks and 3D models. We propose Audio-Lip Memory that brings in visual information of the mouth region corresponding to input audio and enforces fine-grained audio-visual coherence.
  arXiv  Detail & Related papers  (2022-11-02T07:17:49Z)
• LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading [24.744371143092614]
  The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos. We propose LipSound2, which consists of an encoder-decoder architecture and location-aware attention mechanism to map face image sequences to mel-scale spectrograms.
  arXiv  Detail & Related papers  (2021-12-09T08:11:35Z)
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision [63.564385139097624]
  We propose a method to learn self-supervised speech representations from the raw audio waveform. We train a raw audio encoder by combining audio-only self-supervision (by predicting informative audio attributes) with visual self-supervision (by generating talking faces from audio) Our results demonstrate the potential of multimodal self-supervision in audiovisual speech for learning good audio representations.
  arXiv  Detail & Related papers  (2020-07-08T14:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.