Text-To-Speech Synthesis In The Wild

• URL: http://arxiv.org/abs/2409.08711v2
• Date: Sun, 01 Jun 2025 09:29:36 GMT
• Title: Text-To-Speech Synthesis In The Wild
• Authors: Jee-weon Jung, Wangyou Zhang, Soumi Maiti, Yihan Wu, Xin Wang, Ji-Hoon Kim, Yuta Matsunaga, Seyun Um, Jinchuan Tian, Hye-jin Shim, Nicholas Evans, Joon Son Chung, Shinnosuke Takamichi, Shinji Watanabe,
• Abstract summary: We introduce the TTS In the Wild dataset, created through a fully automated pipeline applied to the VoxCeleb1 dataset.<n>It comprises two training sets: TITW-Hard, derived from the transcription, segmentation, and selection of raw VoxCeleb1 data, and TITW-Easy, which incorporates additional enhancement and data selection based on DNSMOS.
• Score: 76.71096751337888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional Text-to-Speech (TTS) systems rely on studio-quality speech recorded in controlled settings.a Recently, an effort known as noisy-TTS training has emerged, aiming to utilize in-the-wild data. However, the lack of dedicated datasets has been a significant limitation. We introduce the TTS In the Wild (TITW) dataset, which is publicly available, created through a fully automated pipeline applied to the VoxCeleb1 dataset. It comprises two training sets: TITW-Hard, derived from the transcription, segmentation, and selection of raw VoxCeleb1 data, and TITW-Easy, which incorporates additional enhancement and data selection based on DNSMOS. State-of-the-art TTS models achieve over 3.0 UTMOS score with TITW-Easy, while TITW-Hard remains difficult showing UTMOS below 2.8.

Related papers

• NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech [0.0]
  NonverbalTTS (NVTTS) is a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories.<n>We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators.
  arXiv  Detail & Related papers  (2025-07-17T14:17:40Z)
• Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM [48.71951982716363]
  Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems. We propose Hard- Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data.
  arXiv  Detail & Related papers  (2024-11-20T09:49:37Z)
• SpoofCeleb: Speech Deepfake Detection and SASV In The Wild [76.71096751337888]
  SpoofCeleb is a dataset designed for Speech Deepfake Detection (SDD) and Spoofing-robust Automatic Speaker Verification (SASV) We utilize source data from real-world conditions and spoofing attacks generated by Text-To-Speech (TTS) systems also trained on the same real-world data. SpoofCeleb comprises over 2.5 million utterances from 1,251 unique speakers, collected under natural, real-world conditions.
  arXiv  Detail & Related papers  (2024-09-18T23:17:02Z)
• Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? [49.42189569058647]
  Two-pass direct speech-to-speech translation (S2ST) models decompose the task into speech-to-text translation (S2TT) and text-to-speech (TTS) In this paper, we introduce a composite S2ST model named ComSpeech, which can seamlessly integrate any pretrained S2TT and TTS models into a direct S2ST model. We also propose a novel training method ComSpeech-ZS that solely utilizes S2TT and TTS data.
  arXiv  Detail & Related papers  (2024-06-11T14:17:12Z)
• Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study [44.07589545984369]
  We propose a fully unsupervised method for building TTS, including automatic data selection and pre-training/fine-tuning strategies. We show how careful selection of data, yet smaller amounts, can improve the efficiency of TTS system. Our objective evaluation shows 3.9% character error rate (CER), while the groundtruth has 1.3% CER.
  arXiv  Detail & Related papers  (2023-01-22T10:41:58Z)
• Generating Synthetic Speech from SpokenVocab for Speech Translation [18.525896864903416]
  Training end-to-end speech translation systems requires sufficiently large-scale data. One practical solution is to convert machine translation data (MT) to ST data via text-to-speech (TTS) systems. We propose a simple, scalable and effective data augmentation technique, i.e., SpokenVocab, to convert MT data to ST data on-the-fly.
  arXiv  Detail & Related papers  (2022-10-15T03:07:44Z)
• Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech [88.22544315633687]
  Polyphone disambiguation aims to capture accurate pronunciation knowledge from natural text sequences for reliable Text-to-speech systems. We propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary. Experimental results in three languages show that our model outperforms several strong baseline models in terms of pronunciation accuracy.
  arXiv  Detail & Related papers  (2022-06-05T10:50:34Z)
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation [76.13334392868208]
  Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue.
  arXiv  Detail & Related papers  (2022-04-06T17:59:22Z)
• Textless Speech-to-Speech Translation on Real Data [49.134208897722246]
  We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language. We tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data.
  arXiv  Detail & Related papers  (2021-12-15T18:56:35Z)
• ESPnet2-TTS: Extending the Edge of TTS Research [62.92178873052468]
  ESPnet2-TTS is an end-to-end text-to-speech (E2E-TTS) toolkit. New features include: on-the-fly flexible pre-processing, joint training with neural vocoders, and state-of-the-art TTS models with extensions like full-band E2E text-to-waveform modeling.
  arXiv  Detail & Related papers  (2021-10-15T03:27:45Z)
• KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset [4.542831770689362]
  This paper introduces a high-quality open-source speech synthesis dataset for Kazakh, a low-resource language spoken by over 13 million people worldwide. The dataset consists of about 91 hours of transcribed audio recordings spoken by two professional speakers. It is the first publicly available large-scale dataset developed to promote Kazakh text-to-speech applications in both academia and industry.
  arXiv  Detail & Related papers  (2021-04-17T05:49:57Z)
• Proteno: Text Normalization with Limited Data for Fast Deployment in Text to Speech Systems [15.401574286479546]
  Text Normalization (TN) systems for Text-to-Speech (TTS) on new languages is hard. We propose a novel architecture to facilitate it for multiple languages while using data less than 3% of the size of the data used by the state of the art results on English. We publish the first results on TN for TTS in Spanish and Tamil and also demonstrate that the performance of the approach is comparable with the previous work done on English.
  arXiv  Detail & Related papers  (2021-04-15T21:14:28Z)
• Data Processing for Optimizing Naturalness of Vietnamese Text-to-speech System [0.7160601421935839]
  We aim to optimize the naturalness of TTS system on found data using a novel data processing method. We showed that an end-to-end TTS achieved a mean opinion score (MOS) of 4.1 compared to 4.3 of natural speech.
  arXiv  Detail & Related papers  (2020-04-20T20:11:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.