Targeted Speaker Poisoning Framework in Zero-Shot Text-to-Speech

• URL: http://arxiv.org/abs/2603.07551v1
• Date: Sun, 08 Mar 2026 09:29:55 GMT
• Title: Targeted Speaker Poisoning Framework in Zero-Shot Text-to-Speech
• Abstract summary: Zero-shot Text-to-Speech (TTS) voice cloning poses severe privacy risks.<n>We formalize this task as Speech Generation Speaker Poisoning (SGSP)<n>We modify trained models to prevent the generation of specific identities while preserving utility for other speakers.
• Score: 42.343691065979215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot Text-to-Speech (TTS) voice cloning poses severe privacy risks, demanding the removal of specific speaker identities from trained TTS models. Conventional machine unlearning is insufficient in this context, as zero-shot TTS can dynamically reconstruct voices from just reference prompts. We formalize this task as Speech Generation Speaker Poisoning (SGSP), in which we modify trained models to prevent the generation of specific identities while preserving utility for other speakers. We evaluate inference-time filtering and parameter-modification baselines across 1, 15, and 100 forgotten speakers. Performance is assessed through the trade-off between utility (WER) and privacy, quantified using AUC and Forget Speaker Similarity (FSSIM). We achieve strong privacy for up to 15 speakers but reveal scalability limits at 100 speakers due to increased identity overlap. Our study thus introduces a novel problem and evaluation framework toward further advances in generative voice privacy.

Related papers

• Hearing the Whispers: Black-Box Membership Inference Attacks on Finetuned TTS Models [23.501699222486412]
  Existing black-box membership inference attacks (MIAs) face unique challenges when adapted to Text-to-Speech (TTS) models.<n>We present the first black-box MIA framework explicitly tailored to TTS models at both the speaker and record levels.
  arXiv  Detail & Related papers  (2026-09-01T18:00:45Z)
• Enhancing Speaker Verification with Whispered Speech via Post-Processing [1.8849814100256281]
  We propose a model with a training recipe to obtain more robust representations against whispered speech hindrances.<n>The proposed system employs an encoder--decoder structure built atop a fine-tuned speaker verification backbone.<n>We also offer a summary of the most popular and state-of-the-art speaker verification models in terms of their performance with whispered speech.
  arXiv  Detail & Related papers  (2026-04-22T06:23:40Z)
• Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech [17.20822861589935]
  ZS-TTS has enabled high-fidelity voice synthesis from minimal audio cues, raising significant privacy and ethical concerns.<n>Despite the threats to voice privacy, research to selectively remove the knowledge to replicate unwanted individual voices has not been explored.<n>We propose the first machine unlearning frameworks for ZS-TTS, especially Teacher-Guided Unlearning (TGU)<n>Our proposed methods incorporate randomness to prevent consistent replication of forget speakers' voices, assuring unlearned identities remain untraceable.
  arXiv  Detail & Related papers  (2025-07-27T06:13:58Z)
• Towards Unsupervised Speech Recognition Without Pronunciation Models [57.222729245842054]
  In this article, we tackle the challenge of developing ASR systems without paired speech and text corpora.<n>We experimentally demonstrate that an unsupervised speech recognizer can emerge from joint speech-to-speech and text-to-text masked token-infilling.<n>This innovative model surpasses the performance of previous unsupervised ASR models under the lexicon-free setting.
  arXiv  Detail & Related papers  (2024-06-12T16:30:58Z)
• USAT: A Universal Speaker-Adaptive Text-to-Speech Approach [11.022840133207788]
  challenge of neglecting lifelike speech for unseen, out-of-dataset speakers remains significant and unresolved. Zero-shot approaches suffer from insufficient generalization performance to reproduce the voice of speakers with heavy accents. Few-shot methods can reproduce highly varying accents, bringing a significant storage burden and the risk of overfitting and catastrophic forgetting. Our proposed framework unifies both zero-shot and few-shot speaker adaptation strategies, which we term as "instant" and "fine-grained" adaptations based on their merits.
  arXiv  Detail & Related papers  (2024-04-28T06:50:55Z)
• Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters [47.75276947690528]
  The zero-shot text-to-speech (TTS) method can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method.
  arXiv  Detail & Related papers  (2024-01-10T12:21:21Z)
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech [26.533600745910437]
  We propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. We also propose a new differentiable pruning method that allows the model to automatically learn the thresholds.
  arXiv  Detail & Related papers  (2023-08-28T21:25:05Z)
• Self-Supervised Speech Representations Preserve Speech Characteristics while Anonymizing Voices [15.136348385992047]
  We train several voice conversion models using self-supervised speech representations. Converted voices retain a low word error rate within 1% of the original voice. Experiments on dysarthric speech data show that speech features relevant to articulation, prosody, phonation and phonology can be extracted from anonymized voices.
  arXiv  Detail & Related papers  (2022-04-04T17:48:01Z)
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios [143.47967241972995]
  We develop AdaSpeech 4, a zero-shot adaptive TTS system for high-quality speech synthesis. We model the speaker characteristics systematically to improve the generalization on new speakers. Without any fine-tuning, AdaSpeech 4 achieves better voice quality and similarity than baselines in multiple datasets.
  arXiv  Detail & Related papers  (2022-04-01T13:47:44Z)
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus [10.158584616360669]
  Training a text-to-speech (TTS) model requires a large scale text labeled speech corpus. We propose a transfer learning framework for TTS that utilizes a large amount of unlabeled speech dataset for pre-training.
  arXiv  Detail & Related papers  (2022-03-29T11:26:56Z)
• Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation [59.41186714127256]
  Dysarthric speech reconstruction (DSR) aims to improve the quality of dysarthric speech. Speaker encoder (SE) optimized for speaker verification has been explored to control the speaker identity. We propose a novel multi-task learning strategy, i.e., adversarial speaker adaptation (ASA)
  arXiv  Detail & Related papers  (2022-02-18T08:59:36Z)
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
  One-shot voice conversion can be effectively achieved by speech representation disentanglement. We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
  arXiv  Detail & Related papers  (2021-06-18T13:50:38Z)
• Speaker De-identification System using Autoencoders and Adversarial Training [58.720142291102135]
  We propose a speaker de-identification system based on adversarial training and autoencoders. Experimental results show that combining adversarial learning and autoencoders increase the equal error rate of a speaker verification system.
  arXiv  Detail & Related papers  (2020-11-09T19:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.