Semantic Codebooks as Effective Priors for Neural Speech Compression

• URL: http://arxiv.org/abs/2512.21653v1
• Date: Thu, 25 Dec 2025 12:49:41 GMT
• Title: Semantic Codebooks as Effective Priors for Neural Speech Compression
• Authors: Liuyang Bai, Weiyi Lu, Li Guo,
• Abstract summary: SemDAC is a semantic-aware neural audio that leverages semantic codebooks as effective priors for speech compression.<n>A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks.
• Score: 3.4074476957610074
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Speech codecs are traditionally optimized for waveform fidelity, allocating bits to preserve acoustic detail even when much of it can be inferred from linguistic structure. This leads to inefficient compression and suboptimal performance on downstream recognition tasks. We propose SemDAC, a semantic-aware neural audio codec that leverages semantic codebooks as effective priors for speech compression. In SemDAC, the first quantizer in a residual vector quantization (RVQ) stack is distilled from HuBERT features to produce semantic tokens that capture phonetic content, while subsequent quantizers model residual acoustics. A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks. Despite its simplicity, this design proves highly effective: SemDAC outperforms DAC across perceptual metrics and achieves lower WER when running Whisper on reconstructed speech, all while operating at substantially lower bitrates (e.g., 0.95 kbps vs. 2.5 kbps for DAC). These results demonstrate that semantic codebooks provide an effective inductive bias for neural speech compression, producing compact yet recognition-friendly representations.

Related papers

• Latent-Mark: An Audio Watermark Robust to Neural Resynthesis [62.09761127079914]
  Latent-Mark is the first zero-bit audio watermarking framework designed to survive semantic compression.<n>Our key insight is that robustness to the encode-decode process requires embedding the watermark within the invariant latent space.<n>Our work inspires future research into universal watermarking frameworks capable of maintaining integrity across increasingly complex and diverse generative distortions.
  arXiv  Detail & Related papers  (2026-03-05T15:51:09Z)
• STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs [19.07983030478734]
  STACodec integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1)<n>We propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference.
  arXiv  Detail & Related papers  (2026-02-05T20:36:24Z)
• Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs [29.817724789807457]
  We introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio units.<n>This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy.
  arXiv  Detail & Related papers  (2025-11-20T18:46:15Z)
• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec [83.61175662066364]
  Speech codecs serve as a crucial bridge in unifying speech and text language models.<n>Existing methods face several challenges in semantic encoding.<n>We propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codecs.
  arXiv  Detail & Related papers  (2025-08-04T19:22:14Z)
• Towards Generalized Source Tracing for Codec-Based Deepfake Speech [52.68106957822706]
  We introduce the Semantic-Acoustic Source Tracing Network (SASTNet), which jointly leverages Whisper for semantic feature encoding and Wav2vec2 with AudioMAE for acoustic feature encoding.<n>Our proposed SASTNet achieves state-of-the-art performance on the CoSG test set of the CodecFake+ dataset, demonstrating its effectiveness for reliable source tracing.
  arXiv  Detail & Related papers  (2025-06-08T21:36:10Z)
• FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks [33.022035588157614]
  FocalCodec is an efficient low-bitrate based on focal modulation that utilizes a single binary codebook to compress speech.<n>Demo samples and code are available at https://lucadellalib.io/kbpscodec-web/.
  arXiv  Detail & Related papers  (2025-02-06T19:24:50Z)
• Latent-Domain Predictive Neural Speech Coding [33.458968443594415]
  This paper introduces latent-domain predictive coding into the VQ-VAE framework.<n>We propose the TF-Codec for low-latency neural speech coding in an end-to-end manner.<n>Subjective results on speech datasets show that, with low latency, the proposed TF-Codec at 1 kbps achieves significantly better quality than at 9 kbps.
  arXiv  Detail & Related papers  (2022-07-18T03:18:08Z)
• Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data [145.95460945321253]
  We introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes. The proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training.
  arXiv  Detail & Related papers  (2022-03-31T15:33:56Z)
• Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation [98.05643473345474]
  We propose a novel decoder, termed dynamic neural representational decoder (NRD) As each location on the encoder's output corresponds to a local patch of the semantic labels, in this work, we represent these local patches of labels with compact neural networks. This neural representation enables our decoder to leverage the smoothness prior in the semantic label space, and thus makes our decoder more efficient.
  arXiv  Detail & Related papers  (2021-07-30T04:50:56Z)
• SoundStream: An End-to-End Neural Audio Codec [78.94923131038682]
  We present SoundStream, a novel neural audio system that can efficiently compress speech, music and general audio. SoundStream relies on a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. We are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency.
  arXiv  Detail & Related papers  (2021-07-07T15:45:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.