E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis

• URL: http://arxiv.org/abs/2511.07099v1
• Date: Mon, 10 Nov 2025 13:38:53 GMT
• Title: E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis
• Abstract summary: Malicious exploitation like voice-cloning fraud poses severe security risks.<n>E2E-VGuard is a proactive defense framework for two emerging threats.<n>We employ the encoder ensemble with a feature extractor to protect timbre.<n>We incorporate the psychoacoustic model to ensure perturbative imperceptibility.
• Score: 27.93620774992674
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in speech synthesis technology have enriched our daily lives, with high-quality and human-like audio widely adopted across real-world applications. However, malicious exploitation like voice-cloning fraud poses severe security risks. Existing defense techniques struggle to address the production large language model (LLM)-based speech synthesis. While previous studies have considered the protection for fine-tuning synthesizers, they assume manually annotated transcripts. Given the labor intensity of manual annotation, end-to-end (E2E) systems leveraging automatic speech recognition (ASR) to generate transcripts are becoming increasingly prevalent, e.g., voice cloning via commercial APIs. Therefore, this E2E speech synthesis also requires new security mechanisms. To tackle these challenges, we propose E2E-VGuard, a proactive defense framework for two emerging threats: (1) production LLM-based speech synthesis, and (2) the novel attack arising from ASR-driven E2E scenarios. Specifically, we employ the encoder ensemble with a feature extractor to protect timbre, while ASR-targeted adversarial examples disrupt pronunciation. Moreover, we incorporate the psychoacoustic model to ensure perturbative imperceptibility. For a comprehensive evaluation, we test 16 open-source synthesizers and 3 commercial APIs across Chinese and English datasets, confirming E2E-VGuard's effectiveness in timbre and pronunciation protection. Real-world deployment validation is also conducted. Our code and demo page are available at https://wxzyd123.github.io/e2e-vguard/.

Related papers

• Never Stop Speaking: a Denial-of-Service Attack on End-to-End Speech Language Models [10.297488739717672]
  We propose a perturbation-based DoS attack targeting E2E speech models.<n>Our method optimize imperceptible acoustic perturbations to directly influence the model's autoregressive generation process.<n>Our method achieves stable attack success rate while significantly increasing generation length and GPU resource consumption.
  arXiv  Detail & Related papers  (2026-08-11T02:50:30Z)
• SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models [56.525932945429275]
  Interleaved spoken language models (SLMs) alternately generate text and speech tokens, but decoding at full transformer depth for every step becomes costly.<n>We propose SPAR-K, a modality-aware early exit framework designed to accelerate interleaved SLM inference while preserving perceptual quality.<n>We evaluate our framework using Step-Audio-2-mini and GLM-4-Voice across four datasets spanning reasoning, factual QA, and dialogue tasks.
  arXiv  Detail & Related papers  (2026-03-10T05:39:03Z)
• Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Models [26.648297855855432]
  We design a text-to-audio jailbreak that embeds disallowed directives within a narrative-style audio stream.<n>The attack exploits structural and acoustic properties, thereby circumventing safety mechanisms primarily calibrated for text.<n>Results highlight the need for safety frameworks that jointly reason over linguistic and paralinguistic representations.
  arXiv  Detail & Related papers  (2026-01-30T18:23:02Z)
• Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio [63.18443674004945]
  This work explores a content-centric threat: exploiting TTS systems to produce speech containing harmful content.<n>We present HARMGEN, a suite of five attacks organized into two families that address these challenges.
  arXiv  Detail & Related papers  (2025-11-14T03:00:04Z)
• When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs [1.911526481015]
  Our research introduces WhisperInject, a two-stage adversarial audio attack framework.<n>It can manipulate state-of-the-art audio language models to generate harmful content.<n>Our method uses imperceptible perturbations in audio inputs that remain benign to human listeners.
  arXiv  Detail & Related papers  (2025-08-05T12:14:01Z)
• DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations [62.00227663434538]
  DRVOICE-7B establishes new state-of-the-art (SOTA) on OpenAudioBench and Big Bench Audio benchmarks.<n>This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling.
  arXiv  Detail & Related papers  (2025-06-11T02:57:22Z)
• Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Framework [6.002582335323663]
  We present an adversarial attack targeting the speech input of aligned Multimodal Large Language Models (MLLMs) in a white box scenario.<n>We introduce a novel token level attack that leverages access to the model's speech tokenization to generate adversarial token sequences.<n>Our approach achieves up to 89 percent attack success rate across multiple restricted tasks.
  arXiv  Detail & Related papers  (2025-05-24T20:46:36Z)
• SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis [8.590034271906289]
  Speech synthesis technology has brought great convenience, while the widespread usage of realistic deepfake audio has triggered hazards.<n>Malicious adversaries may unauthorizedly collect victims' speeches and clone a similar voice for illegal exploitation.<n>We propose a framework, textittextbfSafeSpeech, which protects the users' audio before uploading by embedding imperceptible perturbations on original speeches.
  arXiv  Detail & Related papers  (2025-04-14T03:21:23Z)
• Exploiting Vulnerabilities in Speech Translation Systems through Targeted Adversarial Attacks [59.87470192277124]
  This paper explores methods of compromising speech translation systems through imperceptible audio manipulations.<n>We present two innovative approaches: (1) the injection of perturbation into source audio, and (2) the generation of adversarial music designed to guide targeted translation.<n>Our experiments reveal that carefully crafted audio perturbations can mislead translation models to produce targeted, harmful outputs, while adversarial music achieve this goal more covertly.<n>The implications of this research extend beyond immediate security concerns, shedding light on the interpretability and robustness of neural speech processing systems.
  arXiv  Detail & Related papers  (2025-03-02T16:38:16Z)
• One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition [50.055765860343286]
  This paper presents a novel framework for joint speaker diarization and automatic speech recognition. The framework, named SLIDAR, can process arbitrary length inputs and can handle any number of speakers. Experiments performed on monaural recordings from the AMI corpus confirm the effectiveness of the method in both close-talk and far-field speech scenarios.
  arXiv  Detail & Related papers  (2023-10-02T23:03:30Z)
• Transformer-Based Speech Synthesizer Attribution in an Open Set Scenario [16.93803259128475]
  Speech synthesis methods can create realistic-sounding speech, which may be used for fraud, spoofing, and misinformation campaigns. Forensic attribution methods identify the specific speech synthesis method used to create a speech signal. We propose a speech attribution method that generalizes to new synthesizers not seen during training.
  arXiv  Detail & Related papers  (2022-10-14T05:55:21Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks [54.43697805589634]
  We propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) Our model consists of an encoder-decoder architecture that receives raw video as input and generates speech. We show that this model is able to reconstruct speech with remarkable realism for constrained datasets such as GRID.
  arXiv  Detail & Related papers  (2021-04-27T17:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.