What Counts as Real? Speech Restoration and Voice Quality Conversion Pose New Challenges to Deepfake Detection

• URL: http://arxiv.org/abs/2603.14033v1
• Date: Sat, 14 Mar 2026 17:15:31 GMT
• Title: What Counts as Real? Speech Restoration and Voice Quality Conversion Pose New Challenges to Deepfake Detection
• Abstract summary: We show that phonation-modifying voice conversion and speech restoration are treated as out-of-distribution.<n>We Reformulating anti-spoofing as a multi-class problem improves robustness to benign shifts while preserving spoof detection.
• Score: 22.710371114925763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio anti-spoofing systems are typically formulated as binary classifiers distinguishing bona fide from spoofed speech. This assumption fails under layered generative processing, where benign transformations introduce distributional shifts that are misclassified as spoofing. We show that phonation-modifying voice conversion and speech restoration are treated as out-of-distribution despite preserving speaker authenticity. Using a multi-class setup separating bona fide, converted, spoofed, and converted-spoofed speech, we analyse model behaviour through self-supervised learning (SSL) embeddings and acoustic correlates. The benign transformations induce a drift in the SSL space, compressing bona fide and spoofed speech and reducing classifier separability. Reformulating anti-spoofing as a multi-class problem improves robustness to benign shifts while preserving spoof detection, suggesting binary systems model the distribution of raw speech rather than authenticity itself.

Related papers

• A Training-Free Proactive Defense Against Partial Speech Manipulation via Self-Embedding Steganography [34.948149764638806]
  Partial deepfake speech poses a significant challenge to existing deepfake detection systems.<n>We propose audio steganography as a proactive defense against partially deepfaked audio.<n>We demonstrate how existing audio steganography methods can be repurposed to support detection of partial deepfakes.
  arXiv  Detail & Related papers  (2026-08-26T01:45:35Z)
• REIMU: Efficient Heterogeneous Hierarchical Reasoning for SSL-Based Speech Deepfake Detection [6.6933354669653236]
  Self-supervised learning (SSL) has substantially advanced speech deepfake detection.<n>This work investigates the practical effectiveness of recurrent hierarchical reasoning for this task.
  arXiv  Detail & Related papers  (2026-08-01T20:32:46Z)
• Speaker-Invariant Representation Learning for Spoofing Detection via Gradient Reversal and A Variational Information Bottleneck [7.166778372225167]
  We propose a teacher-student framework for speaker-invariant spoofing detection.<n>We leverage a pre-trained speaker recognition teacher to guide a student model via a gradient reversal layer.<n>We show our model achieves a 25.7% relative reduction to the EER compared to the MHFA baseline.
  arXiv  Detail & Related papers  (2026-06-07T15:31:00Z)
• WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling [35.33131758542107]
  WavCube is a compact continuous latent derived from an SSL speech encoder.<n>It simultaneously supports speech understanding, reconstruction, and generation.<n>Trials show it delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence.
  arXiv  Detail & Related papers  (2026-05-07T15:17:24Z)
• VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses [3.348046946735795]
  Recent defenses attempt to prevent unauthorized cloning by embedding protective perturbations into speech.<n>We proposeVocalBridge, a purification framework that learns a latent mapping from perturbed to clean speech in the EnCodec latent space.<n>We show that our approach consistently outperforms existing purification methods in recovering cloneable voices from protected speech.
  arXiv  Detail & Related papers  (2026-01-05T13:43:30Z)
• MARS-Sep: Multimodal-Aligned Reinforced Sound Separation [72.85468563236005]
  MARS-Sep is a reinforcement learning framework for sound separation.<n>It learns a factorized Beta mask policy that is optimized by a clipped trust-region surrogate.<n>Experiments on multiple benchmarks demonstrate consistent gains in Text-, Audio-, and Image-Queried separation.
  arXiv  Detail & Related papers  (2025-10-12T09:05:28Z)
• SEED: Speaker Embedding Enhancement Diffusion Model [27.198463567915386]
  A primary challenge when deploying speaker recognition systems in real-world applications is performance degradation caused by environmental mismatch.<n>We propose a diffusion-based method that takes speaker embeddings extracted from a pre-trained speaker recognition model and generates refined embeddings.<n>Our method can improve recognition accuracy by up to 19.6% over baseline models while retaining performance on conventional scenarios.
  arXiv  Detail & Related papers  (2025-05-22T15:38:37Z)
• Accent conversion using discrete units with parallel data synthesized from controllable accented TTS [56.18382038512251]
  The goal of accent conversion (AC) is to convert speech accents while preserving content and speaker identity. Previous methods either required reference utterances during inference, did not preserve speaker identity well, or used one-to-one systems that could only be trained for each non-native accent. This paper presents a promising AC model that can convert many accents into native to overcome these issues.
  arXiv  Detail & Related papers  (2024-09-30T19:52:10Z)
• Adversarial Text Purification: A Large Language Model Approach for Defense [25.041109219049442]
  Adversarial purification is a defense mechanism for safeguarding classifiers against adversarial attacks. We propose a novel adversarial text purification that harnesses the generative capabilities of Large Language Models. Our proposed method demonstrates remarkable performance over various classifiers, improving their accuracy under the attack by over 65% on average.
  arXiv  Detail & Related papers  (2024-02-05T02:36:41Z)
• Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering [78.2927924732142]
  We propose speaker-invariant clustering (Spin) as a novel self-supervised learning method. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU.
  arXiv  Detail & Related papers  (2023-05-18T15:59:36Z)
• A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units [94.64927912924087]
  Existing systems ignore the correlation between prosody and language content, leading to degradation of naturalness in converted speech. We devise a cascaded modular system leveraging self-supervised discrete speech units as language representation. Experiments show that our system outperforms previous approaches in naturalness, intelligibility, speaker transferability, and prosody transferability.
  arXiv  Detail & Related papers  (2022-11-12T00:54:09Z)
• Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation [60.26511271597065]
  Speech distortions are a long-standing problem that degrades the performance of supervisely trained speech processing models. It is high time that we enhance the robustness of speech processing models to obtain good performance when encountering speech distortions.
  arXiv  Detail & Related papers  (2022-03-30T07:25:52Z)
• Streaming Multi-talker Speech Recognition with Joint Speaker Identification [77.46617674133556]
  SURIT employs the recurrent neural network transducer (RNN-T) as the backbone for both speech recognition and speaker identification. We validate our idea on the Librispeech dataset -- a multi-talker dataset derived from Librispeech, and present encouraging results.
  arXiv  Detail & Related papers  (2021-04-05T18:37:33Z)
• End-to-End Whisper to Natural Speech Conversion using Modified Transformer Network [0.8399688944263843]
  We introduce whisper-to-natural-speech conversion using sequence-to-sequence approach. We investigate different features like Mel frequency cepstral coefficients and smoothed spectral features. The proposed networks are trained end-to-end using supervised approach for feature-to-feature transformation.
  arXiv  Detail & Related papers  (2020-04-20T14:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.