IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS

• URL: http://arxiv.org/abs/2409.05356v2
• Date: Mon, 7 Oct 2024 05:29:01 GMT
• Title: IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS
• Authors: Ashwin Sankar, Srija Anand, Praveen Srinivasa Varadhan, Sherry Thomas, Mehak Singal, Shridhar Kumar, Deovrat Mehendale, Aditi Krishana, Giri Raju, Mitesh Khapra,
• Abstract summary: IndicVoices-R (IV-R) is the largest multilingual Indian TTS dataset derived from an ASR dataset. IV-R matches the quality of gold-standard TTS datasets like LJ,Speech LibriTTS, and IndicTTS. We release the first TTS model for all 22 official Indian languages.
• Score: 0.9092013845117769
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in text-to-speech (TTS) synthesis show that large-scale models trained with extensive web data produce highly natural-sounding output. However, such data is scarce for Indian languages due to the lack of high-quality, manually subtitled data on platforms like LibriVox or YouTube. To address this gap, we enhance existing large-scale ASR datasets containing natural conversations collected in low-quality environments to generate high-quality TTS training data. Our pipeline leverages the cross-lingual generalization of denoising and speech enhancement models trained on English and applied to Indian languages. This results in IndicVoices-R (IV-R), the largest multilingual Indian TTS dataset derived from an ASR dataset, with 1,704 hours of high-quality speech from 10,496 speakers across 22 Indian languages. IV-R matches the quality of gold-standard TTS datasets like LJSpeech, LibriTTS, and IndicTTS. We also introduce the IV-R Benchmark, the first to assess zero-shot, few-shot, and many-shot speaker generalization capabilities of TTS models on Indian voices, ensuring diversity in age, gender, and style. We demonstrate that fine-tuning an English pre-trained model on a combined dataset of high-quality IndicTTS and our IV-R dataset results in better zero-shot speaker generalization compared to fine-tuning on the IndicTTS dataset alone. Further, our evaluation reveals limited zero-shot generalization for Indian voices in TTS models trained on prior datasets, which we improve by fine-tuning the model on our data containing diverse set of speakers across language families. We open-source all data and code, releasing the first TTS model for all 22 official Indian languages.

Related papers

• Bahasa Harmony: A Comprehensive Dataset for Bahasa Text-to-Speech Synthesis with Discrete Codec Modeling of EnGen-TTS [0.0]
  This research introduces a comprehensive Bahasa text-to-speech dataset and a novel TTS model, EnGen-TTS. The proposed EnGen-TTS model performs better than established baselines, achieving a Mean Opinion Score (MOS) of 4.45 $pm$ 0.13. This research marks a significant advancement in Bahasa TTS technology, with implications for diverse language applications.
  arXiv  Detail & Related papers  (2024-10-09T07:01:05Z)
• SpoofCeleb: Speech Deepfake Detection and SASV In The Wild [76.71096751337888]
  SpoofCeleb is a dataset designed for Speech Deepfake Detection (SDD) and Spoofing-robust Automatic Speaker Verification (SASV) We utilize source data from real-world conditions and spoofing attacks generated by Text-To-Speech (TTS) systems also trained on the same real-world data. SpoofCeleb comprises over 2.5 million utterances from 1,251 unique speakers, collected under natural, real-world conditions.
  arXiv  Detail & Related papers  (2024-09-18T23:17:02Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining [65.30528567491984]
  This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language.
  arXiv  Detail & Related papers  (2023-01-30T00:53:50Z)
• Towards Building Text-To-Speech Systems for the Next Billion Users [18.290165216270452]
  We evaluate the choice of acoustic models, vocoders, supplementary loss functions, training schedules, and speaker and language diversity for Dravidian and Indo-Aryan languages. We train and evaluate TTS models for 13 languages and find our models to significantly improve upon existing models in all languages as measured by mean opinion scores.
  arXiv  Detail & Related papers  (2022-11-17T13:59:34Z)
• Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech [37.942466944970704]
  This paper proposes Virtuoso, a massively multilingual speech-text joint semi-supervised learning framework for text-to-speech synthesis (TTS) models. To train a TTS model from various types of speech and text data, different training schemes are designed to handle supervised (TTS and ASR data) and unsupervised (untranscribed speech and unspoken text) datasets. Experimental evaluation shows that multilingual TTS models trained on Virtuoso can achieve significantly better naturalness and intelligibility than baseline ones in seen languages.
  arXiv  Detail & Related papers  (2022-10-27T14:09:48Z)
• ASR2K: Speech Recognition for Around 2000 Languages without Audio [100.41158814934802]
  We present a speech recognition pipeline that does not require any audio for the target language. Our pipeline consists of three components: acoustic, pronunciation, and language models. We build speech recognition for 1909 languages by combining it with Crubadan: a large endangered languages n-gram database.
  arXiv  Detail & Related papers  (2022-09-06T22:48:29Z)
• Towards Building ASR Systems for the Next Billion Users [15.867823754118422]
  We make contributions towards building ASR systems for low resource languages from the Indian subcontinent. First, we curate 17,000 hours of raw speech data for 40 Indian languages. Using this raw speech data we pretrain several variants of wav2vec style models for 40 Indian languages.
  arXiv  Detail & Related papers  (2021-11-06T19:34:33Z)
• KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset [4.542831770689362]
  This paper introduces a high-quality open-source speech synthesis dataset for Kazakh, a low-resource language spoken by over 13 million people worldwide. The dataset consists of about 91 hours of transcribed audio recordings spoken by two professional speakers. It is the first publicly available large-scale dataset developed to promote Kazakh text-to-speech applications in both academia and industry.
  arXiv  Detail & Related papers  (2021-04-17T05:49:57Z)
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition [148.43282526983637]
  We develop LRSpeech, a TTS and ASR system for languages with low data cost. We conduct experiments on an experimental language (English) and a truly low-resource language (Lithuanian) to verify the effectiveness of LRSpeech. We are currently deploying LRSpeech into a commercialized cloud speech service to support TTS on more rare languages.
  arXiv  Detail & Related papers  (2020-08-09T08:16:33Z)
• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus [57.641761472372814]
  CoVoST is a multilingual speech-to-text translation corpus from 11 languages into English. It diversified with over 11,000 speakers and over 60 accents. CoVoST is released under CC0 license and free to use.
  arXiv  Detail & Related papers  (2020-02-04T14:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.