Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation

• URL: http://arxiv.org/abs/2406.17257v1
• Date: Tue, 25 Jun 2024 03:50:54 GMT
• Title: Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation
• Authors: Yingting Li, Ambuj Mehrish, Bryan Chew, Bo Cheng, Soujanya Poria,
• Abstract summary: Different languages have distinct phonetic systems and vary in their prosodic features making it challenging to develop a Text-to-Speech model. We propose to integrate parameter-efficient transfer learning (PETL) methods such as adapters and hypernetwork with TTS architecture for multilingual speech synthesis.
• Score: 27.78435674869292
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Different languages have distinct phonetic systems and vary in their prosodic features making it challenging to develop a Text-to-Speech (TTS) model that can effectively synthesise speech in multilingual settings. Furthermore, TTS architecture needs to be both efficient enough to capture nuances in multiple languages and efficient enough to be practical for deployment. The standard approach is to build transformer based model such as SpeechT5 and train it on large multilingual dataset. As the size of these models grow the conventional fine-tuning for adapting these model becomes impractical due to heavy computational cost. In this paper, we proposes to integrate parameter-efficient transfer learning (PETL) methods such as adapters and hypernetwork with TTS architecture for multilingual speech synthesis. Notably, in our experiments PETL methods able to achieve comparable or even better performance compared to full fine-tuning with only $\sim$2.5\% tunable parameters.The code and samples are available at: https://anonymous.4open.science/r/multilingualTTS-BA4C.

Related papers

• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• A Text-to-Text Model for Multilingual Offensive Language Identification [19.23565690468299]
  This study presents the first pre-trained model with encoder-decoder architecture for offensive language identification with text-to-text transformers (T5) Our pre-trained T5 model outperforms other transformer-based models fine-tuned for offensive language detection, such as fBERT and HateBERT, in multiple English benchmarks. Following a similar approach, we also train the first multilingual pre-trained model for offensive language identification using mT5.
  arXiv  Detail & Related papers  (2023-12-06T09:37:27Z)
• On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model [49.81429697921861]
  We study the interaction between parameter-efficient fine-tuning (PEFT) and cross-lingual tasks in multilingual autoregressive models. We show that prompt tuning is more effective in enhancing the performance of low-resource languages than fine-tuning.
  arXiv  Detail & Related papers  (2023-11-14T00:43:33Z)
• Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment [31.885608173448368]
  Pre-trained vision and language models such as CLIP have witnessed remarkable success in connecting images and texts with a primary focus on English texts. disparities in performance among different languages have been observed due to uneven resource availability. We propose a new parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities.
  arXiv  Detail & Related papers  (2023-05-02T14:09:02Z)
• Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition [62.997667081978825]
  Code-Switching (CS) is referred to the phenomenon of alternately using words and phrases from different languages. We propose a simple yet effective data augmentation in which audio and corresponding labels of different source languages are transcribed. We show that this augmentation can even improve the model's performance on inter-sentential language switches not seen during training by 5,03% WER.
  arXiv  Detail & Related papers  (2022-10-17T12:15:57Z)
• Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval [66.69799641522133]
  State-of-the-art neural (re)rankers are notoriously data hungry. Current approaches typically transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders. We show that two parameter-efficient approaches to cross-lingual transfer, namely Sparse Fine-Tuning Masks (SFTMs) and Adapters, allow for a more lightweight and more effective zero-shot transfer.
  arXiv  Detail & Related papers  (2022-04-05T15:44:27Z)
• Continual Learning in Multilingual NMT via Language-Specific Embeddings [92.91823064720232]
  It consists in replacing the shared vocabulary with a small language-specific vocabulary and fine-tuning the new embeddings on the new language's parallel data. Because the parameters of the original model are not modified, its performance on the initial languages does not degrade.
  arXiv  Detail & Related papers  (2021-10-20T10:38:57Z)
• Lightweight Adapter Tuning for Multilingual Speech Translation [47.89784337058167]
  Adapter modules were recently introduced as an efficient alternative to fine-tuning in NLP. This paper proposes a comprehensive analysis of adapters for multilingual speech translation.
  arXiv  Detail & Related papers  (2021-06-02T20:51:42Z)
• Adapting Monolingual Models: Data can be Scarce when Language Similarity is High [3.249853429482705]
  We investigate the performance of zero-shot transfer learning with as little data as possible. We retrain the lexical layers of four BERT-based models using data from two low-resource target language varieties. With high language similarity, 10MB of data appears sufficient to achieve substantial monolingual transfer performance.
  arXiv  Detail & Related papers  (2021-05-06T17:43:40Z)
• One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech [3.42658286826597]
  We introduce an approach to multilingual speech synthesis which uses the meta-learning concept of contextual parameter generation. Our model is shown to effectively share information across languages and according to a subjective evaluation test, it produces more natural and accurate code-switching speech than the baselines.
  arXiv  Detail & Related papers  (2020-08-03T10:43:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.