BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting

• URL: http://arxiv.org/abs/2502.05729v1
• Date: Sun, 09 Feb 2025 00:15:47 GMT
• Title: BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting
• Abstract summary: This paper introduces BnTTS, the first framework for Bangla speaker adaptation-based TTS.<n>We pre-train BnTTS on 3.85k hours of Bangla speech dataset with corresponding text labels and evaluate performance in both zero-shot and few-shot settings.<n> Empirical evaluations in few-shot settings show that BnTTS significantly improves the naturalness, intelligibility, and speaker fidelity of synthesized Bangla speech.
• Score: 9.335211682615213
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces BnTTS (Bangla Text-To-Speech), the first framework for Bangla speaker adaptation-based TTS, designed to bridge the gap in Bangla speech synthesis using minimal training data. Building upon the XTTS architecture, our approach integrates Bangla into a multilingual TTS pipeline, with modifications to account for the phonetic and linguistic characteristics of the language. We pre-train BnTTS on 3.85k hours of Bangla speech dataset with corresponding text labels and evaluate performance in both zero-shot and few-shot settings on our proposed test dataset. Empirical evaluations in few-shot settings show that BnTTS significantly improves the naturalness, intelligibility, and speaker fidelity of synthesized Bangla speech. Compared to state-of-the-art Bangla TTS systems, BnTTS exhibits superior performance in Subjective Mean Opinion Score (SMOS), Naturalness, and Clarity metrics.

Related papers

• Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis [57.5830191022097]
  A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation.<n>We adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs.<n>Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines.
  arXiv  Detail & Related papers  (2025-11-07T17:07:56Z)
• Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech [2.5964779217812057]
  Flamed-TTS is a novel zero-shot Text-to-Speech framework that emphasizes low computational cost, low latency, and high speech fidelity alongside rich temporal diversity.<n>We show that Flamed-TTS surpasses state-of-the-art models in terms of intelligibility, naturalness, speaker similarity, acoustic characteristics preservation, and dynamic pace.
  arXiv  Detail & Related papers  (2025-10-03T09:36:55Z)
• UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching [17.559310386487493]
  We introduce UmbraTTS, a flow-matching based TTS model that jointly generates both speech and environmental audio.<n>Our model allows fine-grained control over background volume and produces diverse, coherent, and context-aware audio scenes.
  arXiv  Detail & Related papers  (2025-06-11T15:43:08Z)
• Towards Lightweight and Stable Zero-shot TTS with Self-distilled Representation Disentanglement [17.020173869112163]
  We present a lightweight and stable zero-shot Text-To-Speech (TTS) system.<n>We introduce a novel TTS architecture designed to effectively model linguistic content and speaker attributes from source speech and prompt speech.
  arXiv  Detail & Related papers  (2025-01-15T04:17:48Z)
• Cross-Dialect Text-To-Speech in Pitch-Accent Language Incorporating Multi-Dialect Phoneme-Level BERT [29.167336994990542]
  Cross-dialect text-to-speech (CD-TTS) is a task to synthesize learned speakers' voices in non-native dialects. We present a novel TTS model comprising three sub-modules to perform competitively at this task.
  arXiv  Detail & Related papers  (2024-09-11T13:40:27Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters [47.75276947690528]
  The zero-shot text-to-speech (TTS) method can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method.
  arXiv  Detail & Related papers  (2024-01-10T12:21:21Z)
• DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speech [30.110058338155675]
  Cross-lingual text-to-speech (CTTS) is still far from satisfactory as it is difficult to accurately retain the speaker timbres. We propose a novel dual speaker embedding TTS (DSE-TTS) framework for CTTS with authentic speaking style. By combining both embeddings, DSE-TTS significantly outperforms the state-of-the-art SANE-TTS in cross-lingual synthesis.
  arXiv  Detail & Related papers  (2023-06-25T06:46:36Z)
• Any-speaker Adaptive Text-To-Speech Synthesis with Diffusion Models [65.28001444321465]
  Grad-StyleSpeech is an any-speaker adaptive TTS framework based on a diffusion model. It can generate highly natural speech with extremely high similarity to target speakers' voice, given a few seconds of reference speech. It significantly outperforms speaker-adaptive TTS baselines on English benchmarks.
  arXiv  Detail & Related papers  (2022-11-17T07:17:24Z)
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus [10.158584616360669]
  Training a text-to-speech (TTS) model requires a large scale text labeled speech corpus. We propose a transfer learning framework for TTS that utilizes a large amount of unlabeled speech dataset for pre-training.
  arXiv  Detail & Related papers  (2022-03-29T11:26:56Z)
• Textless Speech-to-Speech Translation on Real Data [49.134208897722246]
  We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language. We tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data.
  arXiv  Detail & Related papers  (2021-12-15T18:56:35Z)
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone [0.7927630381442314]
  YourTTS brings the power of a multilingual approach to the task of zero-shot multi-speaker TTS. We achieve state-of-the-art (SOTA) results in zero-shot multi-speaker TTS and results comparable to SOTA in zero-shot voice conversion on the VCTK dataset. It is possible to fine-tune the YourTTS model with less than 1 minute of speech and achieve state-of-the-art results in voice similarity and with reasonable quality.
  arXiv  Detail & Related papers  (2021-12-04T19:50:29Z)
• Learning Speaker Embedding from Text-to-Speech [59.80309164404974]
  We jointly trained end-to-end Tacotron 2 TTS and speaker embedding networks in a self-supervised fashion. We investigated training TTS from either manual or ASR-generated transcripts. Unsupervised TTS embeddings improved EER by 2.06% absolute with regard to i-vectors for the LibriTTS dataset.
  arXiv  Detail & Related papers  (2020-10-21T18:03:16Z)
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation [125.59372403631006]
  We propose a semi-supervised learning approach for multi-speaker text-to-speech (TTS) A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy.
  arXiv  Detail & Related papers  (2020-05-16T15:47:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.