VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference

• URL: http://arxiv.org/abs/2608.08569v1
• Date: Sun, 09 Aug 2026 08:17:18 GMT
• Title: VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
• Abstract summary: VoxZip is a semantic-anchored KV cache compression framework.<n>First stage uses automatic speech recognition (ASR) transcriptions as explicit semantic anchors to temporally align, compress, and fuse audio tokens.<n>Second stage employs a dynamic filtering strategy based on temporally decayed accumulated attention to evict non-essential tokens.
• Score: 53.883871460875234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks. Despite this progress, their long-context inference remains severely bottlenecked by prohibitive KV cache memory demands. Existing text-centric compression methods struggle here, often disrupting speech continuity or discarding crucial semantic cues. To address this, we propose VoxZip, a train-free, two-stage semantic-anchored KV cache compression framework. The first stage uses automatic speech recognition (ASR) transcriptions as explicit semantic anchors to temporally align, compress, and fuse audio tokens, significantly reducing the initial KV cache while elevating token information density. To further improve the compression ratio, the second stage employs a dynamic filtering strategy based on temporally decayed accumulated attention to evict non-essential tokens while mitigating early-token bias. Comprehensive evaluations on Qwen3-Omni across six diverse audio benchmarks demonstrate the superiority of our approach. VoxZip excels in long-audio reasoning and consistently maintains high-fidelity perception on short-form tasks. Notably, it sustains over 90\% of the uncompressed baseline performance even under an aggressive 20x KV cache compression in long-context scenarios. Furthermore, at a 4x compression ratio, VoxZip yields a 1.9x increase in inference throughput alongside a 3.3x reduction in peak memory overhead. Code and models will be available at https://github.com/MM-Speech/VoxZip.

Related papers

• FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference [0.0]
  FreqDepthKV is an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals.<n>A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits.<n>Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets.
  arXiv  Detail & Related papers  (2026-07-07T17:26:28Z)
• MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression [6.5757024278093565]
  Long-context LLM services sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost.<n>MosaicKV is a dynamic two-D compression system for extremely long-context serving.
  arXiv  Detail & Related papers  (2026-07-01T10:44:57Z)
• End-to-End Context Compression at Scale [81.70601323130997]
  Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.<n>Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.<n>Existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2026-06-08T15:43:16Z)
• MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering [75.0394545769057]
  KV-caching stores the Key-Value of the historical tokens via LLM prefill.<n>MuKV is a method that features a multi-grained KV cache compression module and a semi-hierarchical retrieval approach.<n> Experiments on long-streaming VideoQA benchmarks show that MuKV significantly improves answer accuracy, without sacrificing memory and online QA efficiency.
  arXiv  Detail & Related papers  (2026-05-21T10:13:03Z)
• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction [55.026048429595384]
  EchoKV is a flexible KV cache compression scheme that enables on-demand transitions between standard and compressed inference.<n>We introduce a two-stage fine-tuning strategy that allows for rapid, low-cost training.
  arXiv  Detail & Related papers  (2026-03-24T07:58:42Z)
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models [21.542205813984243]
  We present a training-free, audio-visual token-compression framework that optimize multimodal token representation and inference.<n>Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information.<n>For each time window, OmniZip compresses the video tokens using an interleaved-temporal scheme.
  arXiv  Detail & Related papers  (2025-11-18T15:22:32Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving [23.2180736755494]
  Key-Value cache (textttKV textttcache) compression has emerged as a promising technique to optimize Large Language Model (LLM) serving.<n>It primarily decreases the memory consumption of textttKV textttcache to reduce the computation cost.<n>Despite the development of many compression algorithms, their applications in production environments are still not prevalent.
  arXiv  Detail & Related papers  (2025-03-31T12:23:31Z)
• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression [25.190765258589707]
  We present RocketKV, a training-free KV cache compression strategy containing two consecutive stages.<n>In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens.<n>In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention.
  arXiv  Detail & Related papers  (2025-02-19T19:12:46Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity [24.118503938098307]
  textscPoD allocates memory according to token importance.<n>textscPoD reduces KV cache memory usage by up to 35% without compromising performance.
  arXiv  Detail & Related papers  (2024-12-03T08:29:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.