Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment

• URL: http://arxiv.org/abs/2606.12940v1
• Date: Thu, 11 Jun 2026 06:06:02 GMT
• Title: Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment
• Abstract summary: Self-guidance is used to align the decoder's internal feature manifold when processing both the quantized tokens and their original continuous embeddings.<n>This requires minimal training overhead and no inference-time changes.<n>applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance.
• Score: 15.70133983324942
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural speech codecs based on Vector-Quantized VAEs (VQ-VAEs) are core audio tokenizers for speech LLMs, yet their reconstruction fidelity is bottlenecked by quantization error. Modifying the quantizer or increasing model capacity are common fixes, but they complicate downstream language modeling. Our core idea is to align the decoder's internal feature manifolds when processing both the quantized tokens and their original continuous embeddings, using a lightweight feature-mapping loss. This requires minimal training overhead and no inference-time changes. Applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance. Notably, it enables a 4x codebook reduction without fidelity loss, which downstream TTS experiments show significantly improves LLM-based synthesis by simplifying the token modeling space. Multiple statistical observations and visualizations corroborate the enhanced internal manifold alignment in the decoder. Extensive experiments confirm its generality across various inductive biases. Self-guidance thus establishes an efficient, broadly applicable method for high-fidelity neural audio coding.

Related papers

• NAC: Neural Action Codec for Vision-Language-Action Models [2.8241645653117042]
  Vision-language--action (VLA) models rely on discrete action tokenizers to bridge continuous robot control and autoregressive sequence modeling.<n>We introduce the Neural Action Codec (NAC), which treats short robot action trajectories as multi-channel 1D signals and compresses them using a multi-scale RVQGAN architecture.<n>NAC achieves lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates.
  arXiv  Detail & Related papers  (2026-06-19T12:24:49Z)
• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs [50.3757247538081]
  We retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline.<n>We decompose each encoder's contribution into two axes, Capacity and Necessity.<n>Per-encoder pre-projector effective rank explains the residual score variation.
  arXiv  Detail & Related papers  (2026-06-02T16:46:42Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec [55.40419731151658]
  Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility.<n>In this work, we demonstrate that self-supervised reconstruction (SSRR) loss fundamentally improves training and performance.<n>Our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost.
  arXiv  Detail & Related papers  (2026-03-06T04:13:06Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• Switchcodec: Adaptive residual-expert sparse quantization for high-fidelity neural audio coding [11.19956590509655]
  SwitchCodec is a neural audio based on Residual Experts Vector Quantization (REVQ)<n> REVQ combines a shared quantizer with dynamically routed expert quantizers that are activated according to the input audio.<n>SwitchCodec surpasses existing baselines on both objective metrics and subjective listening tests.
  arXiv  Detail & Related papers  (2026-01-28T08:26:20Z)
• HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling [6.313337261965531]
  We introduce HH-Codec, a neural codecs that achieves extreme compression at 24 tokens per second for 24 kHz audio.<n>Our approach involves a carefully designed Vector Quantization space for Spoken Language Modeling, optimizing compression efficiency while minimizing information loss.<n> HH-Codec achieves state-of-the-art performance in speech reconstruction with an ultra-low bandwidth of 0.3 kbps.
  arXiv  Detail & Related papers  (2025-07-25T02:44:30Z)
• MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation [19.998635838159217]
  MagiCodec is a novel single-layer, streaming Transformer-based audio.<n>We analytically derive the effect of noise injection in the frequency domain, demonstrating its efficacy in atten high-frequency components.<n>We show that MagiCodec surpasses state-of-the-art codecs in both reconstruction quality and downstream tasks.
  arXiv  Detail & Related papers  (2025-05-31T04:31:02Z)
• L3AC: Towards a Lightweight and Lossless Audio Codec [10.903708510237875]
  We introduce L3AC, a lightweight neural audio that addresses challenges by leveraging a single quantizer and a highly efficient architecture.<n>L3AC explores streamlined convolutional networks and local Transformer modules, alongside TConv--a novel structure designed to capture acoustic variations across multiple temporal scales.
  arXiv  Detail & Related papers  (2025-04-07T11:34:39Z)
• Efficient Evaluation of Quantization-Effects in Neural Codecs [4.897318643396687]
  Training neural codecs requires techniques to allow a non-zero gradient across the quantizer.<n>This paper proposes an efficient evaluation framework for neural codecs using simulated data.<n>We validate our findings against an internal neural audio gradient and against the state-of-the-art descript-audio-codec.
  arXiv  Detail & Related papers  (2025-02-07T09:11:19Z)
• High Fidelity Neural Audio Compression [92.4812002532009]
  We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary.
  arXiv  Detail & Related papers  (2022-10-24T17:52:02Z)
• Denoising Diffusion Error Correction Codes [92.10654749898927]
  Recently, neural decoders have demonstrated their advantage over classical decoding techniques. Recent state-of-the-art neural decoders suffer from high complexity and lack the important iterative scheme characteristic of many legacy decoders. We propose to employ denoising diffusion models for the soft decoding of linear codes at arbitrary block lengths.
  arXiv  Detail & Related papers  (2022-09-16T11:00:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.