Leveraging the Interplay Between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation

• URL: http://arxiv.org/abs/2404.02592v1
• Date: Wed, 3 Apr 2024 09:17:38 GMT
• Title: Leveraging the Interplay Between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation
• Authors: Yejin Jeon, Yunsu Kim, Gary Geunbae Lee,
• Abstract summary: We propose a novel framework that incorporates comprehensive modeling of both syntactic and acoustic cues that are associated with pausing patterns. Remarkably, our framework possesses the capability to consistently generate natural speech even for considerably more extended and intricate out-of-domain (OOD) sentences.
• Score: 6.225927189801006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contemporary neural speech synthesis models have indeed demonstrated remarkable proficiency in synthetic speech generation as they have attained a level of quality comparable to that of human-produced speech. Nevertheless, it is important to note that these achievements have predominantly been verified within the context of high-resource languages such as English. Furthermore, the Tacotron and FastSpeech variants show substantial pausing errors when applied to the Korean language, which affects speech perception and naturalness. In order to address the aforementioned issues, we propose a novel framework that incorporates comprehensive modeling of both syntactic and acoustic cues that are associated with pausing patterns. Remarkably, our framework possesses the capability to consistently generate natural speech even for considerably more extended and intricate out-of-domain (OOD) sentences, despite its training on short audio clips. Architectural design choices are validated through comparisons with baseline models and ablation studies using subjective and objective metrics, thus confirming model performance.

Related papers

• Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models [55.898594710420326]
  We propose a novel spontaneous speech synthesis system based on language models. Fine-grained prosody modeling is introduced to enhance the model's ability to capture subtle prosody variations in spontaneous speech.
  arXiv  Detail & Related papers  (2024-07-18T13:42:38Z)
• TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation [97.54885207518946]
  We introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion. We propose two separated encoders to preserve the speaker's voice characteristics and isochrony from the source speech during the translation process. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.
  arXiv  Detail & Related papers  (2024-05-28T04:11:37Z)
• Self-Supervised Models of Speech Infer Universal Articulatory Kinematics [44.27187669492598]
  We show "inference of articulatory kinematics" as fundamental property of SSL models. We also show that this abstraction is largely overlapping across the language of the data used to train the model. We show that with simple affine transformations, Acoustic-to-Articulatory inversion (AAI) is transferrable across speakers, even across genders, languages, and dialects.
  arXiv  Detail & Related papers  (2023-10-16T19:50:01Z)
• M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval [56.49878599920353]
  This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval. For non-English image-speech retrieval, we outperform the current state-of-the-art performance by a wide margin both when training separate models for each language, and with a single model which processes speech in all three languages.
  arXiv  Detail & Related papers  (2022-11-02T14:54:45Z)
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis [23.17929822987861]
  StyleTTS is a style-based generative model for parallel TTS that can synthesize diverse speech with natural prosody from a reference speech utterance. Our method significantly outperforms state-of-the-art models on both single and multi-speaker datasets.
  arXiv  Detail & Related papers  (2022-05-30T21:34:40Z)
• Cross-lingual Low Resource Speaker Adaptation Using Phonological Features [2.8080708404213373]
  We train a language-agnostic multispeaker model conditioned on a set of phonologically derived features common across different languages. With as few as 32 and 8 utterances of target speaker data, we obtain high speaker similarity scores and naturalness comparable to the corresponding literature.
  arXiv  Detail & Related papers  (2021-11-17T12:33:42Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Towards Modelling Coherence in Spoken Discourse [48.80477600384429]
  Coherence in spoken discourse is dependent on the prosodic and acoustic patterns in speech. We model coherence in spoken discourse with audio-based coherence models.
  arXiv  Detail & Related papers  (2020-12-31T20:18:29Z)
• Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis [76.39883780990489]
  We analyze the behavior of non-autoregressive TTS models under different prosody-modeling settings. We propose a hierarchical architecture, in which the prediction of phoneme-level prosody features are conditioned on the word-level prosody features.
  arXiv  Detail & Related papers  (2020-11-12T16:16:41Z)
• Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit [39.258370942013165]
  A prosody learning mechanism is proposed to model the prosody of speech based on TTS system. A novel self-attention structure, named as local attention, is proposed to lift this restriction of input text length. Experiments on English and Mandarin show that speech with more satisfactory prosody has obtained in our model.
  arXiv  Detail & Related papers  (2020-08-13T02:54:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.