ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

• URL: http://arxiv.org/abs/2609.11642v1
• Date: Thu, 10 Sep 2026 14:49:54 GMT
• Title: ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
• Abstract summary: ZipCodec is a streaming neural speech operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms.<n>Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture.<n>ZipCodec achieves real-time single-stream inference on a consumer-grade CPU.
• Score: 27.32235541083431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

Related papers

• LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs [55.497521037224054]
  We propose LACE, a dynamic frame rate that applies an independent compression step at each quantization layer.<n>LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task.<n>Our code is released as part of the ESPnet3 synthesis recipe.
  arXiv  Detail & Related papers  (2026-09-15T17:46:51Z)
• PIC: Revisiting INR for Image Coding with Fast Encoding and Sub-Millisecond Decoding [74.2282307661978]
  Implicit neural representation (INR) has achieved remarkable progress in novel view synthesis and image/video coding in recent years.<n>In this work, we propose a feedforward INR image coding architecture, Practical INR Image Codec (PIC), that computes all the necessary information for INR network in a single forward pass.
  arXiv  Detail & Related papers  (2026-09-08T16:52:00Z)
• Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec [55.40419731151658]
  Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility.<n>In this work, we demonstrate that self-supervised reconstruction (SSRR) loss fundamentally improves training and performance.<n>Our JHCodec achieves state-of-the-art performance while maintaining minimal latency and reduced training cost.
  arXiv  Detail & Related papers  (2026-03-06T04:13:06Z)
• U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation [71.59514998928833]
  U-Codec achieves high-fidelity reconstruction and fast speech generation at an extremely low frame-rate of 5Hz.<n>We apply U-Codec into a large language model (LLM)-based auto-regressive TTS model.
  arXiv  Detail & Related papers  (2025-10-19T05:09:20Z)
• FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation [27.32235541083431]
  FocalCodec-Stream is a hybrid that compresses speech into a single binary codebook at 0.55 - 0.80 kbps with a theoretical latency of 80 ms.<n> Experiments show that FocalCodec-Stream outperforms existing streamable codecs at comparables.
  arXiv  Detail & Related papers  (2025-09-19T17:57:13Z)
• Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates [1.445167946386569]
  We show that Finite Scalar Quantization (FSQ) encodes baked-in redundancy which produces an encoding which is robust when transmitted through noisy channels.<n>We demonstrate that FSQ has vastly superior bit-level perturbation by comparing the performance of RVQ and FSQ codecs when simulating the transmission of code sequences through a noisy channel.
  arXiv  Detail & Related papers  (2025-09-11T15:39:59Z)
• NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference [19.201753265782685]
  Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens.<n>Existing audio codecs often operate at high frame rates, leading to slow training and inference, particularly for autoregressive models.<n>We introduce NanoCodec, a state-of-the-art audio that achieves high-quality compression at just 12.5 frames per second (FPS)
  arXiv  Detail & Related papers  (2025-08-07T20:20:32Z)
• Embedding Compression Distortion in Video Coding for Machines [67.97469042910855]
  Currently, video transmission serves not only the Human Visual System (HVS) for viewing but also machine perception for analysis.<n>We propose a Compression Distortion Embedding (CDRE) framework, which extracts machine-perception-related distortion representation and embeds it into downstream models.<n>Our framework can effectively boost the rate-task performance of existing codecs with minimal overhead in terms of execution time, and number of parameters.
  arXiv  Detail & Related papers  (2025-03-27T13:01:53Z)
• SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound [40.810505707522324]
  SemantiCodec is designed to compress audio into fewer than a hundred tokens per second across diverse audio types.<n>We show that SemantiCodec significantly outperforms the state-of-the-art Descript on reconstruction quality.<n>Our results also suggest that SemantiCodec contains significantly richer semantic information than all evaluated state-of-the-art audio codecs.
  arXiv  Detail & Related papers  (2024-04-30T22:51:36Z)
• High Fidelity Neural Audio Compression [92.4812002532009]
  We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary.
  arXiv  Detail & Related papers  (2022-10-24T17:52:02Z)
• SoundStream: An End-to-End Neural Audio Codec [78.94923131038682]
  We present SoundStream, a novel neural audio system that can efficiently compress speech, music and general audio. SoundStream relies on a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. We are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency.
  arXiv  Detail & Related papers  (2021-07-07T15:45:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.