Your Voice Cloning System is Secretly a Voice Anonymizer

• URL: http://arxiv.org/abs/2608.27360v1
• Date: Thu, 27 Aug 2026 16:59:58 GMT
• Title: Your Voice Cloning System is Secretly a Voice Anonymizer
• Abstract summary: Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality.<n>We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining.
• Score: 6.490209902892897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER $\approx$ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.

Related papers

• KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026 [61.29502937013759]
  Cross-lingual voice cloning aims to generate speech in a target language while preserving speaker identity from a source-language reference.<n>Key challenge is maintaining intelligibility and naturalness in the presence of accent variation and domain-specific vocabulary.<n>We build on a multilingual text-to-speech model, FishAudio-S2-Pro, and introduce language tag prompting to improve language control and reduce accent leakage.
  arXiv  Detail & Related papers  (2026-06-05T13:09:21Z)
• X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning [62.740127542449166]
  X-Voice is a multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages.<n>X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation.
  arXiv  Detail & Related papers  (2026-05-07T02:57:53Z)
• One Voice, Many Tongues: Cross-Lingual Voice Cloning for Scientific Speech [0.24870920787157705]
  Preserving a speaker's voice identity while generating speech in a different language remains a fundamental challenge in spoken language technology.<n>In this paper, we evaluate several state-of-the-art voice cloning models for cross-lingual speech generation of scientific texts in Arabic, Chinese, and French.
  arXiv  Detail & Related papers  (2026-04-28T21:47:52Z)
• PolyVoice: Language Models for Speech to Speech Translation [50.31000706309143]
  PolyVoice is a language model-based framework for speech-to-speech translation (S2ST) We use discretized speech units, which are generated in a fully unsupervised way. For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model.
  arXiv  Detail & Related papers  (2023-06-05T15:53:15Z)
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling [92.55131711064935]
  We propose a cross-lingual neural language model, VALL-E X, for cross-lingual speech synthesis. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. It can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment.
  arXiv  Detail & Related papers  (2023-03-07T14:31:55Z)
• ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech [58.93395189153713]
  We extend the pretraining method for cross-lingual multi-speaker speech synthesis tasks. We propose a speech-text joint pretraining framework, where we randomly mask the spectrogram and the phonemes. Our model shows great improvements over speaker-embedding-based multi-speaker TTS methods.
  arXiv  Detail & Related papers  (2022-11-07T13:35:16Z)
• Improve Cross-lingual Voice Cloning Using Low-quality Code-switched Data [11.18504333789534]
  We propose to use low-quality code-switched found data from the non-target speakers to achieve cross-lingual voice cloning for the target speakers. Experiments show that our proposed method can generate high-quality code-switched speech in the target voices in terms of both naturalness and speaker consistency.
  arXiv  Detail & Related papers  (2021-10-14T08:16:06Z)
• Latent linguistic embedding for cross-lingual text-to-speech and voice conversion [44.700803634034486]
  Cross-lingual speech generation is the scenario in which speech utterances are generated with the voices of target speakers in a language not spoken by them originally. We show that our method not only creates cross-lingual VC with high speaker similarity but also can be seamlessly used for cross-lingual TTS without having to perform any extra steps.
  arXiv  Detail & Related papers  (2020-10-08T01:25:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.