KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution

• URL: http://arxiv.org/abs/2505.00497v1
• Date: Thu, 01 May 2025 12:56:17 GMT
• Title: KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution
• Authors: Antoni Bigata, Rodrigo Mira, Stella Bounareli, Michał Stypułkowski, Konstantinos Vougioukas, Stavros Petridis, Maja Pantic,
• Abstract summary: Lip synchronization presents significant new challenges such as expression leakage from the input video.<n>We present KeySync, a two-stage framework that succeeds in solving the issue of temporal consistency.<n>We show that KeySync state-of-the-art results in lip reconstruction and cross-synchronization, improving visual quality and reducing expression leakage according to LipLeak.
• Score: 32.124841838431166
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Lip synchronization, known as the task of aligning lip movements in an existing video with new input audio, is typically framed as a simpler variant of audio-driven facial animation. However, as well as suffering from the usual issues in talking head generation (e.g., temporal consistency), lip synchronization presents significant new challenges such as expression leakage from the input video and facial occlusions, which can severely impact real-world applications like automated dubbing, but are often neglected in existing works. To address these shortcomings, we present KeySync, a two-stage framework that succeeds in solving the issue of temporal consistency, while also incorporating solutions for leakage and occlusions using a carefully designed masking strategy. We show that KeySync achieves state-of-the-art results in lip reconstruction and cross-synchronization, improving visual quality and reducing expression leakage according to LipLeak, our novel leakage metric. Furthermore, we demonstrate the effectiveness of our new masking approach in handling occlusions and validate our architectural choices through several ablation studies. Code and model weights can be found at https://antonibigata.github.io/KeySync.

Related papers

• Removing Averaging: Personalized Lip-Sync Driven Characters Based on Identity Adapter [10.608872317957026]
  "lip averaging" phenomenon occurs when a model fails to preserve subtle facial details when dubbing unseen in-the-wild videos. We propose UnAvgLip, which extracts identity embeddings from reference videos to generate highly faithful facial sequences.
  arXiv  Detail & Related papers  (2025-03-09T02:36:31Z)
• SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion [78.77211425667542]
  SayAnything is a conditional video diffusion framework that directly synthesizes lip movements from audio input.<n>Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation.
  arXiv  Detail & Related papers  (2025-02-17T07:29:36Z)
• MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling [12.438835523353347]
  Existing approaches face a trilemma: diffusion-based methods achieve high visual fidelity but suffer from prohibitive computational costs.<n>We present MuseTalk, a novel two-stage training framework that resolves this trade-off through latent space optimization and data sampling strategy.<n>MuseTalk establishes an effective audio-visual feature fusion framework in the latent space, delivering 30 FPS output at 256*256 resolution on an NVIDIA V100 GPU.
  arXiv  Detail & Related papers  (2024-10-14T03:22:26Z)
• High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Model [89.29655924125461]
  We propose a novel landmark-based diffusion model for talking face generation. We first establish the less ambiguous mapping from audio to landmark motion of lip and jaw. Then, we introduce an innovative conditioning module called TalkFormer to align the synthesized motion with the motion represented by landmarks.
  arXiv  Detail & Related papers  (2024-08-10T02:58:28Z)
• ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer [87.32518573172631]
  ReSyncer fuses motion and appearance with unified training. It supports fast personalized fine-tuning, video-driven lip-syncing, the transfer of speaking styles, and even face swapping.
  arXiv  Detail & Related papers  (2024-08-06T16:31:45Z)
• SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space [13.59798532129008]
  We propose an innovative unified framework, SwapTalk, which accomplishes both face swapping and lip synchronization tasks in the same latent space. We introduce a novel identity consistency metric to more comprehensively assess the identity consistency over time series in generated facial videos. Experimental results on the HDTF demonstrate that our method significantly surpasses existing techniques in video quality, lip synchronization accuracy, face swapping fidelity, and identity consistency.
  arXiv  Detail & Related papers  (2024-05-09T09:22:09Z)
• Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization [67.88493779080882]
  Video moment localization aims to retrieve the target segment of an untrimmed video according to the natural language query. Recent works contrast the cross-modality similarities driven by reconstructing masked queries. We propose a novel proposed counterfactual cross-modality reasoning method.
  arXiv  Detail & Related papers  (2023-08-10T15:45:45Z)
• Audio-driven Talking Face Generation with Stabilized Synchronization Loss [60.01529422759644]
  Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality. We first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization.
  arXiv  Detail & Related papers  (2023-07-18T15:50:04Z)
• End to End Lip Synchronization with a Temporal AutoEncoder [95.94432031144716]
  We study the problem of syncing the lip movement in a video with the audio stream. Our solution finds an optimal alignment using a dual-domain recurrent neural network. As an application, we demonstrate our ability to robustly align text-to-speech generated audio with an existing video stream.
  arXiv  Detail & Related papers  (2022-03-30T12:00:18Z)
• Data standardization for robust lip sync [10.235718439446044]
  Existing lip sync methods fall short of being robust in the wild. One important cause could be distracting factors on the visual input side, making extracting lip motion information difficult. This paper proposes a data standardization pipeline to standardize the visual input for lip sync.
  arXiv  Detail & Related papers  (2022-02-13T04:09:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.