WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

• URL: http://arxiv.org/abs/2608.22704v2
• Date: Sat, 29 Aug 2026 17:03:06 GMT
• Title: WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
• Abstract summary: Waxing-and-Waning KV cache classifies KV-heads into anchor, tidal, and fixed roles via offline calibration.<n>On Libri-Speech with two 3B backbones, WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate.
• Score: 60.6692467625441
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads keep all audio KV on GPU and yield a decode-time signal of which audio region each token is read from; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.

Related papers

• VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference [53.883871460875234]
  VoxZip is a semantic-anchored KV cache compression framework.<n>First stage uses automatic speech recognition (ASR) transcriptions as explicit semantic anchors to temporally align, compress, and fuse audio tokens.<n>Second stage employs a dynamic filtering strategy based on temporally decayed accumulated attention to evict non-essential tokens.
  arXiv  Detail & Related papers  (2026-08-09T08:17:18Z)
• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• WorldKV: Efficient World Memory with World Retrieval and Compression [56.57011243315561]
  We propose WorldKV, a training-free framework with two components: World Retrieval and World Compression.<n>On Matrix-Game-2.0 and LingBot- World-Fast, WorldKV matches or exceeds full- KV memory fidelity at roughly 2x the throughput, and is competitive with memory-trained baselines without any fine-tuning.
  arXiv  Detail & Related papers  (2026-05-21T16:55:04Z)
• MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering [75.0394545769057]
  KV-caching stores the Key-Value of the historical tokens via LLM prefill.<n>MuKV is a method that features a multi-grained KV cache compression module and a semi-hierarchical retrieval approach.<n> Experiments on long-streaming VideoQA benchmarks show that MuKV significantly improves answer accuracy, without sacrificing memory and online QA efficiency.
  arXiv  Detail & Related papers  (2026-05-21T10:13:03Z)
• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference [24.571166055469508]
  We present VeriCache, the first inference framework that ensures the same output as full- KV-cache decoding.<n>We show that VeriCache achieves up to 4X higher throughput than full- KV-cache.
  arXiv  Detail & Related papers  (2026-05-17T19:18:39Z)
• RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache [28.54642982960947]
  Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth.<n>Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation.<n>In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme.
  arXiv  Detail & Related papers  (2026-05-08T15:15:06Z)
• Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs [6.222287867011644]
  We propose MorphKV, an inference-time technique that maintains a constant-sized KV cache while preserving accuracy.<n>Unlike retention or lossy compression, MorphKV iteratively refines the KV cache via lightweight updates guided by attention patterns of recent tokens.<n>Our studies show 52.9$%$ memory savings and 18.2$%$ higher accuracy on average compared to state-of-the-art prior works.
  arXiv  Detail & Related papers  (2025-03-02T18:12:50Z)
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads [22.462489968597]
  Caching all Key and Value states across all attention heads consumes substantial memory. We introduce DuoAttention, a framework that only applies a full KV cache to retrieval heads while using a light-weight, constant-length KV cache for streaming heads. Our method significantly reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [82.08922896531618]
  We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) We conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens.
  arXiv  Detail & Related papers  (2023-10-03T05:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.