Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec

• URL: http://arxiv.org/abs/2603.05887v1
• Date: Fri, 06 Mar 2026 04:13:06 GMT
• Title: Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec
• Abstract summary: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility.<n>In this work, we demonstrate that self-supervised reconstruction (SSRR) loss fundamentally improves training and performance.<n>Our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost.
• Score: 55.40419731151658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results using only a single GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. As a result, our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost. We open-source the full implementation, training pipeline, and demo on Github https://github.com/jhcodec843/jhcodec.

Related papers

• CodecArena: Codec Quality Assessment via Visual Reinforcement Learning [70.59130074737446]
  We propose CodecArena, the first vision-language framework for video coding quality assessment.<n>CodecArena casts evaluation as source-conditioned comparative reasoning between a reference and its reconstructions.<n>Its facet-anchored reward uses automatically derived facet directions as weak anchors, rather than human per-facet labels.
  arXiv  Detail & Related papers  (2026-08-10T05:37:06Z)
• Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment [15.70133983324942]
  Self-guidance is used to align the decoder's internal feature manifold when processing both the quantized tokens and their original continuous embeddings.<n>This requires minimal training overhead and no inference-time changes.<n>applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance.
  arXiv  Detail & Related papers  (2026-06-11T06:06:02Z)
• Improving Reconstruction of Representation Autoencoder [52.817427902597416]
  We propose LV-RAE, a representation autoencoder that augments semantic features with missing low-level information.<n>Our experiments demonstrate that LV-RAE significantly improves reconstruction fidelity while preserving the semantic abstraction.
  arXiv  Detail & Related papers  (2026-02-09T13:12:35Z)
• FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation [27.32235541083431]
  FocalCodec-Stream is a hybrid that compresses speech into a single binary codebook at 0.55 - 0.80 kbps with a theoretical latency of 80 ms.<n> Experiments show that FocalCodec-Stream outperforms existing streamable codecs at comparables.
  arXiv  Detail & Related papers  (2025-09-19T17:57:13Z)
• SIEDD: Shared-Implicit Encoder with Discrete Decoders [36.705337163276255]
  Implicit Neural Representations (INRs) offer exceptional fidelity for video compression by learning per-video optimized functions.<n>Existing attempts to accelerate INR encoding often sacrifice reconstruction quality or crucial coordinate-level control.<n>We introduce SIEDD, a novel architecture that fundamentally accelerates INR encoding without these compromises.
  arXiv  Detail & Related papers  (2025-06-29T19:39:43Z)
• MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation [19.998635838159217]
  MagiCodec is a novel single-layer, streaming Transformer-based audio.<n>We analytically derive the effect of noise injection in the frequency domain, demonstrating its efficacy in atten high-frequency components.<n>We show that MagiCodec surpasses state-of-the-art codecs in both reconstruction quality and downstream tasks.
  arXiv  Detail & Related papers  (2025-05-31T04:31:02Z)
• Improving the Diffusability of Autoencoders [54.920783089085035]
  Latent diffusion models have emerged as the leading approach for generating high-quality images and videos.<n>We perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces.<n>We hypothesize that this high-frequency component interferes with the coarse-to-fine nature of the diffusion synthesis process and hinders the generation quality.
  arXiv  Detail & Related papers  (2025-02-20T18:45:44Z)
• Epsilon-VAE: Denoising as Visual Decoding [61.29255979767292]
  We propose denoising as decoding, shifting from single-step reconstruction to iterative refinement.<n>Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image.<n>By adopting iterative reconstruction through diffusion, our autoencoder, namely Epsilon-VAE, achieves high reconstruction quality.
  arXiv  Detail & Related papers  (2024-10-05T08:27:53Z)
• Boosting Neural Representations for Videos with a Conditional Decoder [28.073607937396552]
  Implicit neural representations (INRs) have emerged as a promising approach for video storage and processing. This paper introduces a universal boosting framework for current implicit video representation approaches.
  arXiv  Detail & Related papers  (2024-02-28T08:32:19Z)
• An Efficient Implicit Neural Representation Image Codec Based on Mixed Autoregressive Model for Low-Complexity Decoding [43.43996899487615]
  Implicit Neural Representation (INR) for image compression is an emerging technology that offers two key benefits compared to cutting-edge autoencoder models. We introduce a new Mixed AutoRegressive Model (MARM) to significantly reduce the decoding time for the current INR. MARM includes our proposed AutoRegressive Upsampler (ARU) blocks, which are highly efficient, and ARM from previous work to balance decoding time and reconstruction quality.
  arXiv  Detail & Related papers  (2024-01-23T09:37:58Z)
• CNeRV: Content-adaptive Neural Representation for Visual Data [54.99373641890767]
  We propose Neural Visual Representation with Content-adaptive Embedding (CNeRV), which combines the generalizability of autoencoders with the simplicity and compactness of implicit representation. We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images) With the same latent code length and similar model size, CNeRV outperforms autoencoders on reconstruction of both seen and unseen images.
  arXiv  Detail & Related papers  (2022-11-18T18:35:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.