Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detection

• URL: http://arxiv.org/abs/2603.16914v1
• Date: Tue, 10 Mar 2026 09:38:03 GMT
• Title: Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detection
• Abstract summary: We propose a hierarchy-aware representation learning framework for speech deepfake detection.<n>Our method achieves relative EER reductions of 46.2% on ASVspoof 2019 and 13.9% on ASVspoof5 over strong baselines.
• Score: 10.201174843022425
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Neural audio codecs discretize speech via residual vector quantization (RVQ), forming a coarse-to-fine hierarchy across quantizers. While codec models have been explored for representation learning, their discrete structure remains underutilized in speech deepfake detection. In particular, different quantization levels capture complementary acoustic cues, where early quantizers encode coarse structure and later quantizers refine residual details that reveal synthesis artifacts. Existing systems either rely on continuous encoder features or ignore this quantizer-level hierarchy. We propose a hierarchy-aware representation learning framework that models quantizer-level contributions through learnable global weighting, enabling structured codec representations aligned with forensic cues. Keeping the speech encoder backbone frozen and updating only 4.4% additional parameters, our method achieves relative EER reductions of 46.2% on ASVspoof 2019 and 13.9% on ASVspoof5 over strong baselines.

Related papers

• Streaming Neural Speech Codecs through Time-Invariant Representations [18.03681751870144]
  TiCodec introduces a factorized representation that separates time-varying speech content from time-invariant information.<n>We investigate the nature of the information captured by TIRE representations and their suitability for low-latency speech processing.<n>We propose Dual-TIRE, a multi-level architecture that exploits the complementarity of different encoder layers.
  arXiv  Detail & Related papers  (2026-07-06T15:57:49Z)
• Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment [15.70133983324942]
  Self-guidance is used to align the decoder's internal feature manifold when processing both the quantized tokens and their original continuous embeddings.<n>This requires minimal training overhead and no inference-time changes.<n>applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance.
  arXiv  Detail & Related papers  (2026-06-11T06:06:02Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• Spectrogram Patch Codec: A 2D Block-Quantized VQ-VAE and HiFi-GAN for Neural Speech Coding [0.0]
  We present a neural speech that challenges the need for complex residual vector quantization stacks by introducing a simpler, single-stage quantization approach.<n>Our method operates directly on the mel-spectrogram, treating it as a 2D data and quantizing non-overlapping 4x4 patches into a single, shared codebook.<n>This patchwise design simplifies the architecture, enables low-latency streaming, and yields a discrete latent grid.
  arXiv  Detail & Related papers  (2025-09-02T12:14:41Z)
• HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling [6.313337261965531]
  We introduce HH-Codec, a neural codecs that achieves extreme compression at 24 tokens per second for 24 kHz audio.<n>Our approach involves a carefully designed Vector Quantization space for Spoken Language Modeling, optimizing compression efficiency while minimizing information loss.<n> HH-Codec achieves state-of-the-art performance in speech reconstruction with an ultra-low bandwidth of 0.3 kbps.
  arXiv  Detail & Related papers  (2025-07-25T02:44:30Z)
• L3AC: Towards a Lightweight and Lossless Audio Codec [10.903708510237875]
  We introduce L3AC, a lightweight neural audio that addresses challenges by leveraging a single quantizer and a highly efficient architecture.<n>L3AC explores streamlined convolutional networks and local Transformer modules, alongside TConv--a novel structure designed to capture acoustic variations across multiple temporal scales.
  arXiv  Detail & Related papers  (2025-04-07T11:34:39Z)
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling [63.8735398698683]
  A crucial component of language models is the tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens.<n>We introduce WavTokenizer, which offers several advantages over previous SOTA acoustic models in the audio domain.<n>WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information.
  arXiv  Detail & Related papers  (2024-08-29T13:43:36Z)
• High Fidelity Neural Audio Compression [92.4812002532009]
  We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary.
  arXiv  Detail & Related papers  (2022-10-24T17:52:02Z)
• Voice Activity Detection for Transient Noisy Environment Based on Diffusion Nets [13.558688470594674]
  We address voice activity detection in acoustic environments of transients and stationary noises. We exploit unique spatial patterns of speech and non-speech audio frames by independently learning their underlying geometric structure. A deep neural network is trained to separate speech from non-speech frames.
  arXiv  Detail & Related papers  (2021-06-25T17:05:26Z)
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling [61.351967629600594]
  This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity.
  arXiv  Detail & Related papers  (2020-09-06T13:01:06Z)
• Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding [59.48857453699463]
  In sequence-to-sequence learning, the decoder relies on the attention mechanism to efficiently extract information from the encoder. Recent work has proposed to use representations from different encoder layers for diversified levels of information. We propose layer-wise multi-view decoding, where for each decoder layer, together with the representations from the last encoder layer, which serve as a global view, those from other encoder layers are supplemented for a stereoscopic view of the source sequences.
  arXiv  Detail & Related papers  (2020-05-16T20:00:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.