Continual Speaker Identity Unlearning with Minimal Interference

• URL: http://arxiv.org/abs/2605.25962v1
• Date: Mon, 25 May 2026 15:40:04 GMT
• Title: Continual Speaker Identity Unlearning with Minimal Interference
• Abstract summary: Machine unlearning removes designated concepts or knowledge from pre-trained models.<n>Existing methods, however, quietly assume all unlearning requests arrive at once.<n>We show this assumption breaks state-of-the-art methods, reintroducing the very privacy risk unlearning was meant to eliminate.
• Score: 28.890263474409696
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Machine unlearning removes designated concepts or knowledge from pre-trained models. Recent work has extended this paradigm to speaker identity unlearning in zero-shot text-to-speech (ZS-TTS), the task of selectively erasing a model's ability to replicate a speaker's voice. Existing methods, however, quietly assume all unlearning requests arrive at once; an unrealistic assumption, since privacy-motivated removals arrive sequentially over time. We show this assumption breaks state-of-the-art methods: unlearning each new speaker fully revives previously unlearned speakers, reintroducing the very privacy risk unlearning was meant to eliminate. We present Cumulative ORThogonal Identity Suppression (CORTIS), the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines Fisher-information-based parameter masking, which localizes updates to speaker-relevant weights, with orthogonal projection against subspaces spanned by prior unlearning updates. With VoiceBox, CORTIS unlearns each requested speaker while keeping previously unlearned speakers forgotten across long request sequences, substantially outperforming sequential application of prior methods. The demo is available at https://cumulativeortis.github.io/ .

Related papers

• HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding [54.365959989048406]
  Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments.<n>But their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear.<n>We introduce HEAR, a benchmark diagnosing the foundational capabilities of speaker-attributed reasoning.<n>We present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives.
  arXiv  Detail & Related papers  (2026-08-29T08:01:20Z)
• UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction [7.775050285048427]
  Speech assistant is driving artificial propagation toward human-like conversational systems.<n>Front-end components, such as voice activity detection (VAD) and turn-taking detection (TD) are crucial for speech assistant.<n>We propose the first unified audio frontend LLM (UAF) tailored for fullgressive speech systems.
  arXiv  Detail & Related papers  (2026-04-21T08:24:55Z)
• Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions [52.45054413627452]
  TPI-Train is a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling.<n>TPI-Bench is a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy.
  arXiv  Detail & Related papers  (2026-04-19T10:03:42Z)
• Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech [17.20822861589935]
  ZS-TTS has enabled high-fidelity voice synthesis from minimal audio cues, raising significant privacy and ethical concerns.<n>Despite the threats to voice privacy, research to selectively remove the knowledge to replicate unwanted individual voices has not been explored.<n>We propose the first machine unlearning frameworks for ZS-TTS, especially Teacher-Guided Unlearning (TGU)<n>Our proposed methods incorporate randomness to prevent consistent replication of forget speakers' voices, assuring unlearned identities remain untraceable.
  arXiv  Detail & Related papers  (2025-07-27T06:13:58Z)
• Continuously Learning New Words in Automatic Speech Recognition [56.972851337263755]
  We propose a self-supervised continual learning approach for Automatic Speech Recognition.<n>We use a memory-enhanced ASR model from the literature to decode new words from the slides.<n>We show that with this approach, we obtain increasing performance on the new words when they occur more frequently.
  arXiv  Detail & Related papers  (2024-01-09T10:39:17Z)
• Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering [78.2927924732142]
  We propose speaker-invariant clustering (Spin) as a novel self-supervised learning method. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU.
  arXiv  Detail & Related papers  (2023-05-18T15:59:36Z)
• End-to-End Diarization for Variable Number of Speakers with Local-Global Networks and Discriminative Speaker Embeddings [66.50782702086575]
  We present an end-to-end deep network model that performs meeting diarization from single-channel audio recordings. The proposed system is designed to handle meetings with unknown numbers of speakers, using variable-number permutation-invariant cross-entropy based loss functions.
  arXiv  Detail & Related papers  (2021-05-05T14:55:29Z)
• Continual Speaker Adaptation for Text-to-Speech Synthesis [2.3224617218247126]
  In this paper, we look at TTS modeling from a continual learning perspective. The goal is to add new speakers without forgetting previous speakers. We exploit two well-known techniques for continual learning namely experience replay and weight regularization.
  arXiv  Detail & Related papers  (2021-03-26T15:14:20Z)
• Voice Cloning: a Multi-Speaker Text-to-Speech Synthesis Approach based on Transfer Learning [0.802904964931021]
  The proposed approach has the goal to overcome these limitations trying to obtain a system which is able to model a multi-speaker acoustic space. This allows the generation of speech audio similar to the voice of different target speakers, even if they were not observed during the training phase.
  arXiv  Detail & Related papers  (2021-02-10T18:43:56Z)
• End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification [45.38809571153867]
  We propose the End-to-End Neural Diarization (EEND) in which a neural network directly outputs speaker diarization results. By feeding multi-speaker recordings with corresponding speaker segment labels, our model can be easily adapted to real conversations.
  arXiv  Detail & Related papers  (2020-02-24T14:53:32Z)
• Multi-task self-supervised learning for Robust Speech Recognition [75.11748484288229]
  This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. We employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks.
  arXiv  Detail & Related papers  (2020-01-25T00:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.