KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026

• URL: http://arxiv.org/abs/2606.07240v1
• Date: Fri, 05 Jun 2026 13:09:21 GMT
• Title: KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026
• Abstract summary: Cross-lingual voice cloning aims to generate speech in a target language while preserving speaker identity from a source-language reference.<n>Key challenge is maintaining intelligibility and naturalness in the presence of accent variation and domain-specific vocabulary.<n>We build on a multilingual text-to-speech model, FishAudio-S2-Pro, and introduce language tag prompting to improve language control and reduce accent leakage.
• Score: 61.29502937013759
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cross-lingual voice cloning aims to generate speech in a target language while preserving speaker identity from a source-language reference. This task is central to speech translation and is the focus of the IWSLT 2026 Cross-Lingual Voice Cloning track. A key challenge is maintaining intelligibility and naturalness in the presence of accent variation and domain-specific vocabulary. We build on a multilingual text-to-speech model, FishAudio-S2-Pro, and introduce language tag prompting to improve language control and reduce accent leakage. We further apply reinforcement learning (RL) fine-tuning for task adaptation and observe improvements in intelligibility. Finally, we propose a reference-conditioned lexical matching method that improves pronunciation of domain-specific terms when lexical overlap is present. Results show that language prompting provides the largest gains, while lexical matching yields consistent improvements on matched subsets.

Related papers

• Your Voice Cloning System is Secretly a Voice Anonymizer [6.490209902892897]
  Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality.<n>We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining.
  arXiv  Detail & Related papers  (2026-08-27T16:59:58Z)
• X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning [62.740127542449166]
  X-Voice is a multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages.<n>X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation.
  arXiv  Detail & Related papers  (2026-05-07T02:57:53Z)
• One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech [0.24870920787157705]
  Preserving a speaker's voice identity while generating speech in a different language remains a fundamental challenge in spoken language technology.<n>In this paper, we evaluate several state-of-the-art voice cloning models for cross-lingual speech generation of scientific texts in Arabic, Chinese, and French.
  arXiv  Detail & Related papers  (2026-04-28T21:47:52Z)
• Language translation, and change of accent for speech-to-speech task using diffusion model [16.436756456803774]
  Speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another.<n>We propose a unified approach for simultaneous speech translation and change of accent.
  arXiv  Detail & Related papers  (2025-05-04T23:23:46Z)
• CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation [25.82932373649325]
  CrossSpeech++ is a method to disentangle language and speaker information.<n>It significantly improves the quality of cross-lingual speech synthesis.<n>We conduct extensive experiments using various metrics, and demonstrate that CrossSpeech++ achieves significant improvements.
  arXiv  Detail & Related papers  (2024-12-28T06:32:49Z)
• ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation [79.66359274050885]
  We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-05-24T07:42:15Z)
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling [92.55131711064935]
  We propose a cross-lingual neural language model, VALL-E X, for cross-lingual speech synthesis. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. It can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment.
  arXiv  Detail & Related papers  (2023-03-07T14:31:55Z)
• ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech [58.93395189153713]
  We extend the pretraining method for cross-lingual multi-speaker speech synthesis tasks. We propose a speech-text joint pretraining framework, where we randomly mask the spectrogram and the phonemes. Our model shows great improvements over speaker-embedding-based multi-speaker TTS methods.
  arXiv  Detail & Related papers  (2022-11-07T13:35:16Z)
• Improve Cross-lingual Voice Cloning Using Low-quality Code-switched Data [11.18504333789534]
  We propose to use low-quality code-switched found data from the non-target speakers to achieve cross-lingual voice cloning for the target speakers. Experiments show that our proposed method can generate high-quality code-switched speech in the target voices in terms of both naturalness and speaker consistency.
  arXiv  Detail & Related papers  (2021-10-14T08:16:06Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.