Transfer Learning from Monolingual ASR to Transcription-free Cross-lingual Voice Conversion

• URL: http://arxiv.org/abs/2009.14668v1
• Date: Wed, 30 Sep 2020 13:44:35 GMT
• Title: Transfer Learning from Monolingual ASR to Transcription-free Cross-lingual Voice Conversion
• Authors: Che-Jui Chang
• Abstract summary: Cross-lingual voice conversion is a task that aims to synthesize target voices with the same content while source and target speakers speak in different languages. In this paper, we focus on knowledge transfer from monolin-gual ASR to cross-lingual VC. We successfully address cross-lingual VC without any transcription or language-specific knowledge for foreign speech.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-lingual voice conversion (VC) is a task that aims to synthesize target voices with the same content while source and target speakers speak in different languages. Its challenge lies in the fact that the source and target data are naturally non-parallel, and it is even difficult to bridge the gaps between languages with no transcriptions provided. In this paper, we focus on knowledge transfer from monolin-gual ASR to cross-lingual VC, in order to address the con-tent mismatch problem. To achieve this, we first train a monolingual acoustic model for the source language, use it to extract phonetic features for all the speech in the VC dataset, and then train a Seq2Seq conversion model to pre-dict the mel-spectrograms. We successfully address cross-lingual VC without any transcription or language-specific knowledge for foreign speech. We experiment this on Voice Conversion Challenge 2020 datasets and show that our speaker-dependent conversion model outperforms the zero-shot baseline, achieving MOS of 3.83 and 3.54 in speech quality and speaker similarity for cross-lingual conversion. When compared to Cascade ASR-TTS method, our proposed one significantly reduces the MOS drop be-tween intra- and cross-lingual conversion.

Related papers

• MulliVC: Multi-lingual Voice Conversion With Cycle Consistency [75.59590240034261]
  MulliVC is a novel voice conversion system that only converts timbre and keeps original content and source language prosody without multi-lingual paired data. Both objective and subjective results indicate that MulliVC significantly surpasses other methods in both monolingual and cross-lingual contexts.
  arXiv  Detail & Related papers  (2024-08-08T18:12:51Z)
• Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech [6.243356997302935]
  We introduce a framework for cross-lingual speech synthesis, which involves an upstream Voice Conversion (VC) model and a downstream Text-To-Speech (TTS) model. In the first two stages, we use a VC model to convert utterances in the target locale to the voice of the target speaker. In the third stage, the converted data is combined with the linguistic features and durations from recordings in the target language, which are then used to train a single-speaker acoustic model.
  arXiv  Detail & Related papers  (2023-09-15T09:03:14Z)
• Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer [53.72998363956454]
  Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy. The scarcity of high-quality speaker-parallel data poses a challenge for learning style transfer during translation. We design an S2ST pipeline with style-transfer capability on the basis of discrete self-supervised speech representations and timbre units.
  arXiv  Detail & Related papers  (2023-09-14T09:52:08Z)
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling [92.55131711064935]
  We propose a cross-lingual neural language model, VALL-E X, for cross-lingual speech synthesis. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. It can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment.
  arXiv  Detail & Related papers  (2023-03-07T14:31:55Z)
• Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation [11.336431583289382]
  This paper presents a method for end-to-end cross-lingual text-to-speech. It aims to preserve the target language's pronunciation regardless of the original speaker's language.
  arXiv  Detail & Related papers  (2022-10-31T12:44:53Z)
• ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion [49.617722668505834]
  We show that our approach permits the application of speech synthesis and voice conversion to improve ASR systems using only one target-language speaker during model training. It is possible to obtain promising ASR training results with our data augmentation method using only a single real speaker in a target language.
  arXiv  Detail & Related papers  (2022-03-29T11:55:30Z)
• On Prosody Modeling for ASR+TTS based Voice Conversion [82.65378387724641]
  In voice conversion, an approach showing promising results in the latest voice conversion challenge (VCC) 2020 is to first use an automatic speech recognition (ASR) model to transcribe the source speech into the underlying linguistic contents. Such a paradigm, referred to as ASR+TTS, overlooks the modeling of prosody, which plays an important role in speech naturalness and conversion similarity. We propose to directly predict prosody from the linguistic representation in a target-speaker-dependent manner, referred to as target text prediction (TTP)
  arXiv  Detail & Related papers  (2021-07-20T13:30:23Z)
• Towards Natural and Controllable Cross-Lingual Voice Conversion Based on Neural TTS Model and Phonetic Posteriorgram [21.652906261475533]
  Cross-lingual voice conversion is a challenging problem due to significant mismatches of the phonetic set and the speech prosody of different languages. We build upon the neural text-to-speech (TTS) model to design a new cross-lingual VC framework named FastSpeech-VC.
  arXiv  Detail & Related papers  (2021-02-03T10:28:07Z)
• How Phonotactics Affect Multilingual and Zero-shot ASR Performance [74.70048598292583]
  A Transformer encoder-decoder model has been shown to leverage multilingual data well in IPA transcriptions of languages presented during training. We replace the encoder-decoder with a hybrid ASR system consisting of a separate AM and LM. We show that the gain from modeling crosslingual phonotactics is limited, and imposing a too strong model can hurt the zero-shot transfer.
  arXiv  Detail & Related papers  (2020-10-22T23:07:24Z)
• Latent linguistic embedding for cross-lingual text-to-speech and voice conversion [44.700803634034486]
  Cross-lingual speech generation is the scenario in which speech utterances are generated with the voices of target speakers in a language not spoken by them originally. We show that our method not only creates cross-lingual VC with high speaker similarity but also can be seamlessly used for cross-lingual TTS without having to perform any extra steps.
  arXiv  Detail & Related papers  (2020-10-08T01:25:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.