Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation

• URL: http://arxiv.org/abs/2005.08024v2
• Date: Tue, 4 Aug 2020 07:55:34 GMT
• Title: Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
• Authors: Tao Tu, Yuan-Jui Chen, Alexander H. Liu, Hung-yi Lee
• Abstract summary: We propose a semi-supervised learning approach for multi-speaker text-to-speech (TTS) A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy.
• Score: 125.59372403631006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, end-to-end multi-speaker text-to-speech (TTS) systems gain success in the situation where a lot of high-quality speech plus their corresponding transcriptions are available. However, laborious paired data collection processes prevent many institutes from building multi-speaker TTS systems of great performance. In this work, we propose a semi-supervised learning approach for multi-speaker TTS. A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. The experiment results demonstrate that with only an hour of paired speech data, no matter the paired data is from multiple speakers or a single speaker, the proposed model can generate intelligible speech in different voices. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy. In addition, our analysis reveals that different speaker characteristics of the paired data have an impact on the effectiveness of semi-supervised TTS.

Related papers

• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation [23.895122319920997]
  We tackle single-channel multi-speaker conversational ST with an end-to-end and multi-task training model. Speaker-Turn Aware Conversational Speech Translation combines automatic speech recognition, speech translation and speaker turn detection. We show that our model outperforms the reference systems on the multi-speaker condition, while attaining comparable performance on the single-speaker condition.
  arXiv  Detail & Related papers  (2023-11-01T17:55:09Z)
• ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech [58.93395189153713]
  We extend the pretraining method for cross-lingual multi-speaker speech synthesis tasks. We propose a speech-text joint pretraining framework, where we randomly mask the spectrogram and the phonemes. Our model shows great improvements over speaker-embedding-based multi-speaker TTS methods.
  arXiv  Detail & Related papers  (2022-11-07T13:35:16Z)
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus [10.158584616360669]
  Training a text-to-speech (TTS) model requires a large scale text labeled speech corpus. We propose a transfer learning framework for TTS that utilizes a large amount of unlabeled speech dataset for pre-training.
  arXiv  Detail & Related papers  (2022-03-29T11:26:56Z)
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech [62.95422526044178]
  We use Model Agnostic Meta-Learning (MAML) as the training algorithm of a multi-speaker TTS model. We show that Meta-TTS can synthesize high speaker-similarity speech from few enrollment samples with fewer adaptation steps than the speaker adaptation baseline.
  arXiv  Detail & Related papers  (2021-11-07T09:53:31Z)
• GC-TTS: Few-shot Speaker Adaptation with Geometric Constraints [36.07346889498981]
  We propose GC-TTS which achieves high-quality speaker adaptation with significantly improved speaker similarity. A TTS model is pre-trained for base speakers with a sufficient amount of data, and then fine-tuned for novel speakers on a few minutes of data with two geometric constraints. The experimental results demonstrate that GC-TTS generates high-quality speech from only a few minutes of training data, outperforming standard techniques in terms of speaker similarity to the target speaker.
  arXiv  Detail & Related papers  (2021-08-16T04:25:31Z)
• GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis [6.632254395574993]
  GANSpeech is a high-fidelity multi-speaker TTS model that adopts the adversarial training method to a non-autoregressive multi-speaker TTS model. In the subjective listening tests, GANSpeech significantly outperformed the baseline multi-speaker FastSpeech and FastSpeech2 models.
  arXiv  Detail & Related papers  (2021-06-29T08:15:30Z)
• Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech [54.75722224061665]
  In this work, we investigate different speaker representations and proposed to integrate pretrained and learnable speaker representations. The FastSpeech 2 model combined with both pretrained and learnable speaker representations shows great generalization ability on few-shot speakers.
  arXiv  Detail & Related papers  (2021-03-06T10:14:33Z)
• Synth2Aug: Cross-domain speaker recognition with TTS synthesized speech [8.465993273653554]
  We investigate the use of a multi-speaker Text-To-Speech system to synthesize speech in support of speaker recognition. We observe on our datasets that TTS synthesized speech improves cross-domain speaker recognition performance. We also explore the effectiveness of different types of text transcripts used for TTS synthesis.
  arXiv  Detail & Related papers  (2020-11-24T00:48:54Z)
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding [61.02342238771685]
  Spoken language understanding requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. Various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. We propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules.
  arXiv  Detail & Related papers  (2020-10-05T19:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.