HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models

• URL: http://arxiv.org/abs/2609.02029v1
• Date: Wed, 02 Sep 2026 02:59:11 GMT
• Title: HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
• Abstract summary: Long-context inference retains a growing key--value (KV) cache during decoding.<n>This bottleneck remains in hybrid language models because their residual global-attention layers can dominate context-dependent cache demand.<n>We introduce HeadWiseKV, a training-free framework that compresses the residual global KV caches of hybrid language models.
• Score: 10.767250624894665
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context inference retains a growing key--value (KV) cache during decoding, which consumes substantial GPU memory and can reduce generation throughput. This bottleneck remains in hybrid language models because their residual global-attention layers can dominate context-dependent cache demand. We study how to allocate this state under an aggregate KV-residency budget. We introduce HeadWiseKV, a training-free framework that compresses the residual global KV caches of hybrid language models while preserving their native local, recurrent, and linear paths. It assigns each physical KV head a static, multilevel history window, making cache demand predictable before serving. We formulate this allocation as a restricted operational rate--distortion problem and propose SeqCalib as the core policy-generation algorithm in HeadWiseKV. SeqCalib processes layers in execution order and conditions each decision on the lower-layer policy used at deployment, thereby accounting for interactions across depth. A grouped-cache runtime materializes the selected policy as actual per-head KV residency rather than a mask over a full cache. We evaluate downstream quality across four hybrid long-context models and study physical residency and serving behavior on Qwen3.6-27B. HeadWiseKV retains near-Full-KV RULER and LoCoMo quality across the evaluated models. In the fixed-model systems study, it reduces sampled peak device memory by 8.59\% at a 112K context length and extends the largest verified successful context from 114K to 161K.

Related papers

• DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference [11.462434448123169]
  We propose a layer-dependent pruning framework that allocates a fixed global KV budget across layers based on sensitivity.<n>Across multiple models and tasks, DepthKV consistently outperforms uniform pruning at the same global pruning ratio.
  arXiv  Detail & Related papers  (2026-04-27T16:15:37Z)
• Kwai Summary Attention Technical Report [69.40814939510126]
  Long-context ability, has become one of the most important direction of next-generation Large Language Models.<n>Standard softmax attention exhibits quadratic time complexity with respect to sequence length.<n>We propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts.
  arXiv  Detail & Related papers  (2026-04-27T12:59:53Z)
• Comparative Characterization of KV Cache Management Strategies for LLM Inference [0.31498833540989407]
  Key-Value caches are essential for Efficient inference with Large Language Models (LLMs)<n>These caches are essential to minimize redundant computation during autoregressive token generation.<n>The growth of KV caches has posed significant system-level challenges.
  arXiv  Detail & Related papers  (2026-04-06T16:00:39Z)
• Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing [29.913403615975174]
  Serving transformer language models with high throughput requires caching Key-Values ( KVs) to avoid redundant computation.<n>The memory footprint of KV caching is significant and heavily impacts serving costs.<n>We propose a simple training approach: random cross-layer attention. During training, layers randomly choose to attend either to their own KV states or those of a preceding layer.
  arXiv  Detail & Related papers  (2026-04-03T14:56:17Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• EpiCache: Episodic KV Cache Management for Long Conversational Question Answering [15.288494370436469]
  We introduce EpiCache, a training-free KV cache management framework for long conversational question answering.<n>EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression.<n>Across three LongConvQA benchmarks, EpiCache improves accuracy by up to 40%, maintains near-full KV accuracy under 4-6x compression, and reduces latency/memory by up to 2.4x/3.5x.
  arXiv  Detail & Related papers  (2025-09-22T06:56:35Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference [6.222836318380985]
  BaKlaVa is a method to allocate optimal memory for individual KV-caches across the model.<n>We evaluate our method on LLaMA-3-8B, and Qwen2.5-7B models.
  arXiv  Detail & Related papers  (2025-02-18T04:08:29Z)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods [61.025422435235456]
  We introduce SCBench, a benchmark for evaluating long-context methods from a KV cachecentric perspective.<n>We provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs and Mamba-Attention hybrids.<n>Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n2) pre-filling perform robustly.
  arXiv  Detail & Related papers  (2024-12-13T17:59:52Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• QAQ: Quality Adaptive Quantization for LLM KV Cache [3.163526369095745]
  A bottleneck in model deployment emerges due to the linear expansion of the Key-Value cache with the context length. We propose QAQ, a Quality Adaptive Quantization scheme for the KV cache.
  arXiv  Detail & Related papers  (2024-03-07T16:42:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.