Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

• URL: http://arxiv.org/abs/2303.17480v1
• Date: Wed, 29 Mar 2023 07:51:07 GMT
• Title: Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
• Authors: Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, Haizhou Li
• Abstract summary: Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. Previous studies revealed the importance of lip-speech synchronization and visual quality. We propose using a lip-reading expert to improve the intelligibility of the generated lip regions.
• Score: 89.07178484337865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. The previous studies revealed the importance of lip-speech synchronization and visual quality. Despite much progress, they hardly focus on the content of lip movements i.e., the visual intelligibility of the spoken words, which is an important aspect of generation quality. To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results. Moreover, to compensate for data scarcity, we train the lip-reading expert in an audio-visual self-supervised manner. With a lip-reading expert, we propose a novel contrastive learning to enhance lip-speech synchronization, and a transformer to encode audio synchronically with video, while considering global temporal dependency of audio. For evaluation, we propose a new strategy with two different lip-reading experts to measure intelligibility of the generated videos. Rigorous experiments show that our proposal is superior to other State-of-the-art (SOTA) methods, such as Wav2Lip, in reading intelligibility i.e., over 38% Word Error Rate (WER) on LRS2 dataset and 27.8% accuracy on LRW dataset. We also achieve the SOTA performance in lip-speech synchronization and comparable performances in visual quality.

Related papers

• Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert [13.60808166889775]
  We introduce a method for speech-driven 3D facial animation to generate accurate lip movements. This loss provides guidance to train the speech-driven 3D facial animators to generate plausible lip motions aligned with the spoken transcripts. We validate the effectiveness of our approach through broad experiments, showing noticeable improvements in lip synchronization and lip readability performance.
  arXiv  Detail & Related papers  (2024-07-01T07:39:28Z)
• Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation [51.92522679353731]
  We propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. We introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance.
  arXiv  Detail & Related papers  (2024-05-07T13:55:50Z)
• Leveraging Visemes for Better Visual Speech Representation and Lip Reading [2.7836084563851284]
  We propose a novel approach that leverages visemes, which are groups of phonetically similar lip shapes, to extract more discriminative and robust video features for lip reading. The proposed method reduces the lip-reading word error rate (WER) by 9.1% relative to the best previous method.
  arXiv  Detail & Related papers  (2023-07-19T17:38:26Z)
• Audio-driven Talking Face Generation with Stabilized Synchronization Loss [60.01529422759644]
  Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality. We first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization.
  arXiv  Detail & Related papers  (2023-07-18T15:50:04Z)
• Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation [58.72068260933836]
  Context-Aware LipSync- framework (CALS) CALS is comprised of an Audio-to-Lip map module and a Lip-to-Face module.
  arXiv  Detail & Related papers  (2023-05-31T04:50:32Z)
• Learning Speaker-specific Lip-to-Speech Generation [28.620557933595585]
  This work aims to understand the correlation/mapping between speech and the sequence of lip movement of individual speakers. We learn temporal synchronization using deep metric learning, which guides the decoder to generate speech in sync with input lip movements. We have trained our model on the Grid and Lip2Wav Chemistry lecture dataset to evaluate single speaker natural speech generation tasks.
  arXiv  Detail & Related papers  (2022-06-04T19:40:02Z)
• Sub-word Level Lip Reading With Visual Attention [88.89348882036512]
  We focus on the unique challenges encountered in lip reading and propose tailored solutions. We obtain state-of-the-art results on the challenging LRS2 and LRS3 benchmarks when training on public datasets. Our best model achieves 22.6% word error rate on the LRS2 dataset, a performance unprecedented for lip reading models.
  arXiv  Detail & Related papers  (2021-10-14T17:59:57Z)
• SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory [61.44510300515693]
  We study the task of simultaneous lip and devise SimulLR, a simultaneous lip Reading transducer with attention-guided adaptive memory. The experiments show that the SimulLR achieves the translation speedup 9.10 times times compared with the state-of-the-art non-simultaneous methods.
  arXiv  Detail & Related papers  (2021-08-31T05:54:16Z)
• Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis [37.37319356008348]
  We explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis.
  arXiv  Detail & Related papers  (2020-05-17T10:29:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.