HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec

• URL: http://arxiv.org/abs/2606.06743v1
• Date: Thu, 04 Jun 2026 21:57:18 GMT
• Title: HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
• Abstract summary: HybridCodec employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream.<n>It shows superior semantic robustness (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all)<n>We demonstrate its specialization in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures with semantic and acoustic disentanglement have emerged. There are two main approaches to introduce semantic information into codec models: one distills semantic information from SSL representations into the first RVQ layer, while the other maintains separate streams for semantic and acoustic features. We propose HybridCodec, a unified architecture that combines both paradigms. It employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream. This design ensures strong disentanglement without requiring an SSL model during inference. HybridCodec shows superior semantic specialization (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all). We demonstrate its robustness in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.

Related papers

• EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement [54.95940885045993]
  EntangleCodec is a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization.<n>It captures linguistic content, speaker identity, emotion, prosody, and acoustic scenes within a compact token stream.<n>It achieves reconstruction quality competitive with specialized codecs, and supports both TTS and TTA generation in a unified framework.
  arXiv  Detail & Related papers  (2026-06-01T18:05:18Z)
• WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling [35.33131758542107]
  WavCube is a compact continuous latent derived from an SSL speech encoder.<n>It simultaneously supports speech understanding, reconstruction, and generation.<n>Trials show it delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence.
  arXiv  Detail & Related papers  (2026-05-07T15:17:24Z)
• Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation [66.53544128707817]
  Cheers is a unified multimodal model that decouples patch-level details from semantic representations.<n> Cheers matches or surpasses advanced UMMs in both visual understanding and generation.
  arXiv  Detail & Related papers  (2026-03-13T08:55:27Z)
• STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecs [19.07983030478734]
  STACodec integrates semantic information from self-supervised learning (SSL) models into the first layer of residual vector quantization (RVQ-1)<n>We propose a semantic pre-distillation (SPD) module, which predicts semantic tokens directly for assignment to the first RVQ layer during inference.
  arXiv  Detail & Related papers  (2026-02-05T20:36:24Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec [83.61175662066364]
  Speech codecs serve as a crucial bridge in unifying speech and text language models.<n>Existing methods face several challenges in semantic encoding.<n>We propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codecs.
  arXiv  Detail & Related papers  (2025-08-04T19:22:14Z)
• HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling [6.313337261965531]
  We introduce HH-Codec, a neural codecs that achieves extreme compression at 24 tokens per second for 24 kHz audio.<n>Our approach involves a carefully designed Vector Quantization space for Spoken Language Modeling, optimizing compression efficiency while minimizing information loss.<n> HH-Codec achieves state-of-the-art performance in speech reconstruction with an ultra-low bandwidth of 0.3 kbps.
  arXiv  Detail & Related papers  (2025-07-25T02:44:30Z)
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models [66.11184017840688]
  We introduce C3LLM, a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities. Our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model.
  arXiv  Detail & Related papers  (2024-05-25T09:10:12Z)
• BEATs: Audio Pre-Training with Acoustic Tokenizers [77.8510930885778]
  Self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. We propose BEATs, an iterative audio pre-training framework to learn Bidirectional representation from Audio Transformers. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model.
  arXiv  Detail & Related papers  (2022-12-18T10:41:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.