Scores Know Bobs Voice: Speaker Impersonation Attack

• URL: http://arxiv.org/abs/2603.02781v1
• Date: Tue, 03 Mar 2026 09:20:48 GMT
• Title: Scores Know Bobs Voice: Speaker Impersonation Attack
• Abstract summary: We propose an inversion-based generative attack framework that aligns the latent space of a synthesis model with the discriminative feature space of SRSs.<n>Experiments show that our method significantly improves query efficiency, achieving competitive attack success rates with on average 10x fewer queries than prior approaches.
• Score: 8.404098071525473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advances in deep learning have enabled the widespread deployment of speaker recognition systems (SRSs), yet they remain vulnerable to score-based impersonation attacks. Existing attacks that operate directly on raw waveforms require a large number of queries due to the difficulty of optimizing in high-dimensional audio spaces. Latent-space optimization within generative models offers improved efficiency, but these latent spaces are shaped by data distribution matching and do not inherently capture speaker-discriminative geometry. As a result, optimization trajectories often fail to align with the adversarial direction needed to maximize victim scores. To address this limitation, we propose an inversion-based generative attack framework that explicitly aligns the latent space of the synthesis model with the discriminative feature space of SRSs. We first analyze the requirements of an inverse model for score-based attacks and introduce a feature-aligned inversion strategy that geometrically synchronizes latent representations with speaker embeddings. This alignment ensures that latent updates directly translate into score improvements. Moreover, it enables new attack paradigms, including subspace-projection-based attacks, which were previously infeasible due to the absence of a faithful feature-to-audio mapping. Experiments show that our method significantly improves query efficiency, achieving competitive attack success rates with on average 10x fewer queries than prior approaches. In particular, the enabled subspace-projection-based attack attains up to 91.65% success using only 50 queries. These findings establish feature-aligned inversion as a key tool for evaluating the robustness of modern SRSs against score-based impersonation threats.

Related papers

• DeepBias: Adaptive In-depth Probing of Social Biases in LVLMs [71.85279836937578]
  We introduce DeepBias, an adaptive framework for the in-depth probing of social biases in LVLMs with carefully designed agents.<n>We build a benchmark named DeepBiasBench using our framework. By employing an ensemble of five diverse state-of-the-art LVLMs as anchors, the benchmark captures vulnerabilities shared across architectures.
  arXiv  Detail & Related papers  (2026-07-13T08:19:09Z)
• Generative Testing of Automated Speech Recognition Systems [0.4914401344126589]
  We introduce GATAS, a black-box testing approach that generates failure inducing inputs.<n>Instead of perturbing waveforms directly, the approach interpolates latent representations to induce transcription errors.<n>We show that GATAS achieves a 98% success rate while producing lower distortion and higher perceptual quality.
  arXiv  Detail & Related papers  (2026-07-10T14:04:28Z)
• Exploiting Neural Audio Codec Latents for Adversarial Audio Attacks [2.3067587087700767]
  We propose a generative attack framework operating in the latent space of a neural audio.<n>A conditional generator synthesizes class-specific perturbations in a single forward pass and decodes them into adversarial waveforms.<n>Our method targeted attack success rates up to 99% with sub-7 ms inference, outperforming generative baselines while reducing latency by 24x.
  arXiv  Detail & Related papers  (2026-06-18T19:40:46Z)
• SEP-Attack: A Simple and Effective Paradigm for Transfer-Based Textual Adversarial Attack [34.576815318831684]
  Transferable adversarial attacks are still under-explored in the text domain.<n>We propose a paradigm for transfer-based textual adversarial attack, named SEP-Attack.<n>Experiments conducted on four datasets and two real-world APIs validate the efficacy of SEP-Attack.
  arXiv  Detail & Related papers  (2026-05-24T09:15:01Z)
• REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations [60.451310947394596]
  We formulate hallucination elicitation as a constrained optimization problem.<n>The goal is to find semantically coherent adversarial prompts that are equivalent to benign user prompts.<n>We propose REALISTA, a realistic latent-space attack framework.
  arXiv  Detail & Related papers  (2026-05-12T23:13:50Z)
• Low Rank Adaptation for Adversarial Perturbation [50.24377354110586]
  Low-Rank Adaptation (LoRA) has significantly improved the training efficiency of Large Language Models (LLMs)<n>Since the generation of adversarial examples is an optimization process analogous to model training, this naturally raises the question: Do adversarial perturbations exhibit a similar low-rank structure?<n>In this paper, we provide both theoretical analysis and extensive empirical investigation across various attack methods, model architectures, and datasets to show that adversarial perturbations indeed possess a low-rank structure.
  arXiv  Detail & Related papers  (2026-04-30T06:38:28Z)
• SPEAR++: Scaling Gradient Inversion via Sparsely-Used Dictionary Learning [48.41770886055744]
  Federated Learning has seen an increased deployment in real-world scenarios recently.<n>The introduction of the so-called gradient inversion attacks has challenged its privacy-preserving properties.<n>We introduce SPEAR, which is based on a theoretical analysis of the gradients of linear layers with ReLU activations.<n>Our new attack, SPEAR++, retains all desirable properties of SPEAR, such as robustness to DP noise and FedAvg aggregation.
  arXiv  Detail & Related papers  (2025-10-28T09:06:19Z)
• Multi-Metric Preference Alignment for Generative Speech Restoration [15.696247605348383]
  We propose a multi-metric preference alignment strategy for generative models.<n>We observe consistent and significant performance gains across three diverse generative paradigms.<n>Our aligned models can serve as powerful ''data annotators'', generating high-quality pseudo-labels.
  arXiv  Detail & Related papers  (2025-08-24T07:05:10Z)
• Sampling-aware Adversarial Attacks Against Large Language Models [52.30089653615172]
  Existing adversarial attacks typically target harmful responses in single-point greedy generations.<n>We show that for the goal of eliciting harmful responses, repeated sampling of model outputs during the attack prompt optimization.<n>We show that integrating sampling into existing attacks boosts success rates by up to 37% and improves efficiency by up to two orders of magnitude.
  arXiv  Detail & Related papers  (2025-07-06T16:13:33Z)
• Improving Black-Box Generative Attacks via Generator Semantic Consistency [51.470649503929344]
  generative attacks produce adversarial examples in a single forward pass at test time.<n>We enforce semantic consistency by aligning the early generator's intermediate features to an EMA teacher.<n>Our approach can be seamlessly integrated into existing generative attacks with consistent improvements in black-box transfer.
  arXiv  Detail & Related papers  (2025-06-23T02:35:09Z)
• AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models [13.807596637437808]
  Recent advancements in large audio-guided models (LALMs) have enabled speech-based user interactions.<n>However, ensuring the safety of LALMs is crucial to prevent risky outputs that may raise societal concerns or AI regulations.
  arXiv  Detail & Related papers  (2024-12-11T18:30:57Z)
• Toward Improving Synthetic Audio Spoofing Detection Robustness via Meta-Learning and Disentangled Training With Adversarial Examples [33.445126880876415]
  We propose a reliable and robust spoofing detection system to filter out spoofing attacks instead of having them reach the automatic speaker verification system. A weighted additive angular margin loss is proposed to address the data imbalance issue, and different margins has been assigned to improve generalization to unseen spoofing attacks. We craft adversarial examples by adding imperceptible perturbations to spoofing speech as a data augmentation strategy, then we use an auxiliary batch normalization to guarantee that corresponding normalization statistics are performed exclusively on the adversarial examples.
  arXiv  Detail & Related papers  (2024-08-23T19:26:54Z)
• Advancing Generalized Transfer Attack with Initialization Derived Bilevel Optimization and Dynamic Sequence Truncation [49.480978190805125]
  Transfer attacks generate significant interest for black-box applications. Existing works essentially directly optimize the single-level objective w.r.t. surrogate model. We propose a bilevel optimization paradigm, which explicitly reforms the nested relationship between the Upper-Level (UL) pseudo-victim attacker and the Lower-Level (LL) surrogate attacker.
  arXiv  Detail & Related papers  (2024-06-04T07:45:27Z)
• Enhancing Adversarial Robustness via Score-Based Optimization [22.87882885963586]
  Adversarial attacks have the potential to mislead deep neural network classifiers by introducing slight perturbations. We introduce a novel adversarial defense scheme named ScoreOpt, which optimize adversarial samples at test-time. Our experimental results demonstrate that our approach outperforms existing adversarial defenses in terms of both performance and robustness speed.
  arXiv  Detail & Related papers  (2023-07-10T03:59:42Z)
• Towards Robust Speech-to-Text Adversarial Attack [78.5097679815944]
  This paper introduces a novel adversarial algorithm for attacking the state-of-the-art speech-to-text systems, namely DeepSpeech, Kaldi, and Lingvo. Our approach is based on developing an extension for the conventional distortion condition of the adversarial optimization formulation. Minimizing over this metric, which measures the discrepancies between original and adversarial samples' distributions, contributes to crafting signals very close to the subspace of legitimate speech recordings.
  arXiv  Detail & Related papers  (2021-03-15T01:51:41Z)
• Temporal Sparse Adversarial Attack on Sequence-based Gait Recognition [56.844587127848854]
  We demonstrate that the state-of-the-art gait recognition model is vulnerable to such attacks. We employ a generative adversarial network based architecture to semantically generate adversarial high-quality gait silhouettes or video frames. The experimental results show that if only one-fortieth of the frames are attacked, the accuracy of the target model drops dramatically.
  arXiv  Detail & Related papers  (2020-02-22T10:08:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.