Continual Speaker Adaptation for Text-to-Speech Synthesis

• URL: http://arxiv.org/abs/2103.14512v1
• Date: Fri, 26 Mar 2021 15:14:20 GMT
• Title: Continual Speaker Adaptation for Text-to-Speech Synthesis
• Authors: Hamed Hemati, Damian Borth
• Abstract summary: In this paper, we look at TTS modeling from a continual learning perspective. The goal is to add new speakers without forgetting previous speakers. We exploit two well-known techniques for continual learning namely experience replay and weight regularization.
• Score: 2.3224617218247126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training a multi-speaker Text-to-Speech (TTS) model from scratch is computationally expensive and adding new speakers to the dataset requires the model to be re-trained. The naive solution of sequential fine-tuning of a model for new speakers can cause the model to have poor performance on older speakers. This phenomenon is known as catastrophic forgetting. In this paper, we look at TTS modeling from a continual learning perspective where the goal is to add new speakers without forgetting previous speakers. Therefore, we first propose an experimental setup and show that serial fine-tuning for new speakers can result in the forgetting of the previous speakers. Then we exploit two well-known techniques for continual learning namely experience replay and weight regularization and we reveal how one can mitigate the effect of degradation in speech synthesis diversity in sequential training of new speakers using these methods. Finally, we present a simple extension to improve the results in extreme setups.

Related papers

• SelectTTS: Synthesizing Anyone's Voice via Discrete Unit-Based Frame Selection [7.6732312922460055]
  We propose SelectTTS, a novel method to select the appropriate frames from the target speaker and decode using frame-level self-supervised learning (SSL) features. We show that this approach can effectively capture speaker characteristics for unseen speakers, and achieves comparable results to other multi-speaker text-to-speech frameworks in both objective and subjective metrics.
  arXiv  Detail & Related papers  (2024-08-30T17:34:46Z)
• Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representations [12.388567657230116]
  We propose a generalizable zero-shot speaker adaptive text-to-speech and voice conversion model. GZS-TV introduces disentangled representation learning for speaker embedding extraction and timbre transformation. Our experiments demonstrate that GZS-TV reduces performance degradation on unseen speakers and outperforms all baseline models in multiple datasets.
  arXiv  Detail & Related papers  (2023-08-24T18:13:10Z)
• Any-speaker Adaptive Text-To-Speech Synthesis with Diffusion Models [65.28001444321465]
  Grad-StyleSpeech is an any-speaker adaptive TTS framework based on a diffusion model. It can generate highly natural speech with extremely high similarity to target speakers' voice, given a few seconds of reference speech. It significantly outperforms speaker-adaptive TTS baselines on English benchmarks.
  arXiv  Detail & Related papers  (2022-11-17T07:17:24Z)
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios [143.47967241972995]
  We develop AdaSpeech 4, a zero-shot adaptive TTS system for high-quality speech synthesis. We model the speaker characteristics systematically to improve the generalization on new speakers. Without any fine-tuning, AdaSpeech 4 achieves better voice quality and similarity than baselines in multiple datasets.
  arXiv  Detail & Related papers  (2022-04-01T13:47:44Z)
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech [62.95422526044178]
  We use Model Agnostic Meta-Learning (MAML) as the training algorithm of a multi-speaker TTS model. We show that Meta-TTS can synthesize high speaker-similarity speech from few enrollment samples with fewer adaptation steps than the speaker adaptation baseline.
  arXiv  Detail & Related papers  (2021-11-07T09:53:31Z)
• A study on the efficacy of model pre-training in developing neural text-to-speech system [55.947807261757056]
  This study aims to understand better why and how model pre-training can positively contribute to TTS system performance. It is found that the TTS system could achieve comparable performance when the pre-training data is reduced to 1/8 of its original size.
  arXiv  Detail & Related papers  (2021-10-08T02:09:28Z)
• GC-TTS: Few-shot Speaker Adaptation with Geometric Constraints [36.07346889498981]
  We propose GC-TTS which achieves high-quality speaker adaptation with significantly improved speaker similarity. A TTS model is pre-trained for base speakers with a sufficient amount of data, and then fine-tuned for novel speakers on a few minutes of data with two geometric constraints. The experimental results demonstrate that GC-TTS generates high-quality speech from only a few minutes of training data, outperforming standard techniques in terms of speaker similarity to the target speaker.
  arXiv  Detail & Related papers  (2021-08-16T04:25:31Z)
• GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis [6.632254395574993]
  GANSpeech is a high-fidelity multi-speaker TTS model that adopts the adversarial training method to a non-autoregressive multi-speaker TTS model. In the subjective listening tests, GANSpeech significantly outperformed the baseline multi-speaker FastSpeech and FastSpeech2 models.
  arXiv  Detail & Related papers  (2021-06-29T08:15:30Z)
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation [63.561944239071615]
  StyleSpeech is a new TTS model which synthesizes high-quality speech and adapts to new speakers. With SALN, our model effectively synthesizes speech in the style of the target speaker even from single speech audio. We extend it to Meta-StyleSpeech by introducing two discriminators trained with style prototypes, and performing episodic training.
  arXiv  Detail & Related papers  (2021-06-06T15:34:11Z)
• Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement [1.7704011486040843]
  We show that one can transfer an existing TTS model for new speakers from the same or a different language using only 20 minutes of data. We first introduce a base multi-lingual Tacotron with language-agnostic input, then demonstrate how transfer learning is done for different scenarios of speaker adaptation.
  arXiv  Detail & Related papers  (2020-11-12T14:05:34Z)
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation [125.59372403631006]
  We propose a semi-supervised learning approach for multi-speaker text-to-speech (TTS) A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy.
  arXiv  Detail & Related papers  (2020-05-16T15:47:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.