PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing

• URL: http://arxiv.org/abs/2604.09111v3
• Date: Tue, 14 Apr 2026 01:51:32 GMT
• Title: PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing
• Abstract summary: This paper proposes a synchronization method for AD processes that paraphrases translated text.<n>We achieve isochrony by paraphrasing the translated text with a language model, ensuring the target speech duration matches that of the source speech.<n>Second, we introduce PS, which employs dynamic time warping (DTW) with local costs of vowel distances measured from training data so that the target text composes vowels with pronunciations similar to source vowels.<n>Third, we extend this approach to PSComet, which jointly considers semantic and phonetic similarity to preserve meaning better.
• Score: 2.374660957323975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, artificial intelligence-based dubbing technology has advanced, enabling automated dubbing (AD) to convert the source speech of a video into target speech in different languages. However, natural AD still faces synchronization challenges such as duration and lip-synchronization (lip-sync), which are crucial for preserving the viewer experience. Therefore, this paper proposes a synchronization method for AD processes that paraphrases translated text, comprising two steps: isochrony for timing constraints and phonetic synchronization (PS) to preserve lip-sync. First, we achieve isochrony by paraphrasing the translated text with a language model, ensuring the target speech duration matches that of the source speech. Second, we introduce PS, which employs dynamic time warping (DTW) with local costs of vowel distances measured from training data so that the target text composes vowels with pronunciations similar to source vowels. Third, we extend this approach to PSComet, which jointly considers semantic and phonetic similarity to preserve meaning better. The proposed methods are incorporated into text-to-speech systems, PS-TTS and PS-Comet TTS. The performance evaluation using Korean and English lip-reading datasets and a voice-actor dubbing dataset demonstrates that both systems outperform TTS without PS on several objective metrics and outperform voice actors in Korean-to-English and English-to-Korean dubbing. We extend the experiments to French, testing all pairs among these languages to evaluate cross-linguistic applicability. Across all language pairs, PS-Comet performed best, balancing lip-sync accuracy with semantic preservation, confirming that PS-Comet achieves more accurate lip-sync with semantic preservation than PS alone.

Related papers

• When Speech Meets Lips: Interpretable Audio-Visual Synchronization for L2 Pronunciation Assessment [12.51525565225002]
  We propose an interpretable audio-visual synchronization framework that explicitly models speech-lip temporal alignment.<n>We interviewed 30 participants, including 10 instructors and 20 students with diverse first-language backgrounds, to assess its effectiveness.
  arXiv  Detail & Related papers  (2026-09-06T19:11:33Z)
• SyncVoice: Towards Video Dubbing with Vision-Augmented Pretrained TTS Model [34.874153953305346]
  Video dubbing aims to generate high-fidelity speech that is precisely temporally aligned with the visual content.<n>Existing methods still suffer from limitations in speech naturalness and audio-visual synchronization.<n>We propose SyncVoice, a vision-augmented video dubbing framework built upon a pretrained text-to-speech (TTS) model.
  arXiv  Detail & Related papers  (2025-11-23T16:51:05Z)
• Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing [15.134076873312809]
  Cross-lingual dubbing system translates speech from one language to another while preserving key characteristics such as duration, speaker identity, and speaking speed.<n>We propose a discrete diffusion-based speech-to-unit translation model with explicit duration control, enabling time-aligned translation.<n>We then synthesize speech based on the predicted units and source identity with a conditional flow matching model.
  arXiv  Detail & Related papers  (2025-05-27T08:43:28Z)
• SyncSpeech: Low-Latency and Efficient Dual-Stream Text-to-Speech based on Temporal Masked Transformer [68.78023656892319]
  This paper presents a dual-stream text-to-speech (TTS) model, SyncSpeech, capable of receiving streaming text input from upstream models while simultaneously generating streaming speech.<n>SyncSpeech has the following advantages: Low latency, as it begins generating streaming speech upon receiving the second text token; High efficiency, as it decodes all speech tokens corresponding to the each arrived text token in one step.
  arXiv  Detail & Related papers  (2025-02-16T12:14:17Z)
• High-Fidelity Simultaneous Speech-To-Speech Translation [75.69884829562591]
  We introduce Hibiki, a decoder-only model for simultaneous speech translation.<n>Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation.
  arXiv  Detail & Related papers  (2025-02-05T17:18:55Z)
• TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation [97.54885207518946]
  We introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion. We propose two separated encoders to preserve the speaker's voice characteristics and isochrony from the source speech during the translation process. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.
  arXiv  Detail & Related papers  (2024-05-28T04:11:37Z)
• Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing [71.02335065794384]
  We propose a model that directly optimize both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.
  arXiv  Detail & Related papers  (2023-02-25T04:23:25Z)
• Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation [37.48971774827332]
  We propose to generate ST tokens out-of-order while remembering how to re-order them later. We examine two variants of such operation sequences which enable generation of monotonic transcriptions and non-monotonic translations.
  arXiv  Detail & Related papers  (2022-11-11T02:29:28Z)
• Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation [71.35243644890537]
  End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions. Existing zero-shot methods fail to align the two modalities of speech and text into a shared semantic space. We propose a novel Discrete Cross-Modal Alignment (DCMA) method that employs a shared discrete vocabulary space to accommodate and match both modalities of speech and text.
  arXiv  Detail & Related papers  (2022-10-18T03:06:47Z)
• VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over [68.22776506861872]
  We formulate a novel task to synthesize speech in sync with a silent pre-recorded video, denoted as automatic voice over (AVO) A natural solution to AVO is to condition the speech rendering on the temporal progression of lip sequence in the video. We propose a novel text-to-speech model that is conditioned on visual input, named VisualTTS, for accurate lip-speech synchronization.
  arXiv  Detail & Related papers  (2021-10-07T11:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.