VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents

• URL: http://arxiv.org/abs/2505.21568v2
• Date: Fri, 30 May 2025 12:11:04 GMT
• Title: VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents
• Authors: Haiyun Li, Zhiyong Wu, Xiaofeng Xie, Jingran Xie, Yaoxun Xu, Hanyang Peng,
• Abstract summary: Voice cloning (VC)-resistant watermarking is an emerging technique for tracing and preventing unauthorized cloning.<n>Existing methods effectively trace traditional VC models by training them on watermarked audio but fail in zero-shot VC scenarios.<n>We propose VoiceMark, the first zero-shot VC-resistant watermarking method that leverages speaker-specific latents as the watermark carrier.
• Score: 11.902170180363473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Voice cloning (VC)-resistant watermarking is an emerging technique for tracing and preventing unauthorized cloning. Existing methods effectively trace traditional VC models by training them on watermarked audio but fail in zero-shot VC scenarios, where models synthesize audio from an audio prompt without training. To address this, we propose VoiceMark, the first zero-shot VC-resistant watermarking method that leverages speaker-specific latents as the watermark carrier, allowing the watermark to transfer through the zero-shot VC process into the synthesized audio. Additionally, we introduce VC-simulated augmentations and VAD-based loss to enhance robustness against distortions. Experiments on multiple zero-shot VC models demonstrate that VoiceMark achieves over 95% accuracy in watermark detection after zero-shot VC synthesis, significantly outperforming existing methods, which only reach around 50%. See our code and demos at: https://huggingface.co/spaces/haiyunli/VoiceMark

Related papers

• De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks [68.41885995006643]
  We study the first systematic evaluation of protective perturbations against voice cloning (VC) under realistic threat models.<n>Our findings reveal that while existing purification methods can neutralize a considerable portion of the protective perturbations, they still lead to distortions in the feature space of VC models.<n>We propose a novel two-stage purification method: (1) Purify the perturbed speech; (2) Refine it using phoneme guidance to align it with the clean speech distribution.
  arXiv  Detail & Related papers  (2025-07-03T13:30:58Z)
• Speech Watermarking with Discrete Intermediate Representations [45.892635912641836]
  We propose a novel speech watermarking framework that injects watermarks into the discrete intermediate representations of speech.<n>DiscreteWM achieves state-of-the-art performance in robustness and imperceptibility, simultaneously.<n>Our flexible frame-wise approach can serve as an efficient solution for both voice cloning detection and information hiding.
  arXiv  Detail & Related papers  (2024-12-18T14:57:06Z)
• AudioMarkBench: Benchmarking Robustness of Audio Watermarking [38.25450275151647]
  We present AudioMarkBench, the first systematic benchmark for evaluating the robustness of audio watermarking against watermark removal and watermark forgery. Our findings highlight the vulnerabilities of current watermarking techniques and emphasize the need for more robust and fair audio watermarking solutions.
  arXiv  Detail & Related papers  (2024-06-11T06:18:29Z)
• Proactive Detection of Voice Cloning with Localized Watermarking [50.13539630769929]
  We present AudioSeal, the first audio watermarking technique designed specifically for localized detection of AI-generated speech. AudioSeal employs a generator/detector architecture trained jointly with a localization loss to enable localized watermark detection up to the sample level. AudioSeal achieves state-of-the-art performance in terms of robustness to real life audio manipulations and imperceptibility based on automatic and human evaluation metrics.
  arXiv  Detail & Related papers  (2024-01-30T18:56:22Z)
• WavMark: Watermarking for Audio Generation [70.65175179548208]
  This paper introduces an innovative audio watermarking framework that encodes up to 32 bits of watermark within a mere 1-second audio snippet. The watermark is imperceptible to human senses and exhibits strong resilience against various attacks. It can serve as an effective identifier for synthesized voices and holds potential for broader applications in audio copyright protection.
  arXiv  Detail & Related papers  (2023-08-24T13:17:35Z)
• Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust [55.91987293510401]
  Watermarking the outputs of generative models is a crucial technique for tracing copyright and preventing potential harm from AI-generated content. We introduce a novel technique called Tree-Ring Watermarking that robustly fingerprints diffusion model outputs. Our watermark is semantically hidden in the image space and is far more robust than watermarking alternatives that are currently deployed.
  arXiv  Detail & Related papers  (2023-05-31T17:00:31Z)
• Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion [42.43123253495082]
  One-shot voice conversion (VC) with only a single target speaker's speech for reference has become a hot research topic. We employ random resampling for pitch and content encoder and use the variational contrastive log-ratio upper bound of mutual information to disentangle speech components. Experiments on the VCTK dataset show the model achieves state-of-the-art performance for one-shot VC in terms of naturalness and intellgibility.
  arXiv  Detail & Related papers  (2022-08-18T10:36:27Z)
• Voicy: Zero-Shot Non-Parallel Voice Conversion in Noisy Reverberant Environments [76.98764900754111]
  Voice Conversion (VC) is a technique that aims to transform the non-linguistic information of a source utterance to change the perceived identity of the speaker. We propose Voicy, a new VC framework particularly tailored for noisy speech. Our method, which is inspired by the de-noising auto-encoders framework, is comprised of four encoders (speaker, content, phonetic and acoustic-ASR) and one decoder.
  arXiv  Detail & Related papers  (2021-06-16T15:47:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.