LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

• URL: http://arxiv.org/abs/2605.06676v1
• Date: Wed, 22 Apr 2026 06:35:23 GMT
• Title: LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
• Abstract summary: We introduce LKV (Learned KV Eviction), which formulates KV compression as an end-to-end differentiable optimization problem.<n>LKV achieves state-of-the-art performance on both LongBench and RULER benchmarks at high compression rates.
• Score: 46.60403085398673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context inference in Large Language Models (LLMs) is bottlenecked by the linear growth of Key-Value (KV) cache memory. Existing KV cache compression paradigms are fundamentally limited by heuristics: heuristic budgeting relies on statistical priors rather than task objectives, causing resource misallocation, while heuristic selection relies on coupled query-key interactions or static inductive biases (e.g., attention sinks). To address this limitation, we introduce LKV (Learned KV Eviction), which formulates KV compression as an end-to-end differentiable optimization problem. LKV integrates LKV-H to learn task-optimized global budgets, and LKV-T to derive intrinsic KV importance without materializing attention matrices. This design bypasses heuristic proxies, strictly aligning compression with task objectives. Extensive evaluations demonstrate that LKV achieves state-of-the-art performance on both LongBench and RULER benchmarks at high compression rates. In particular, on LongBench, LKV achieves near-lossless performance with only 15\% KV cache retention. Crucially, our analysis identifies learned budgeting as the dominant driver of fidelity, demonstrating that data-driven allocation is essential to overcome the limitations of hand-crafted heuristics.

Related papers

• SGD-KV: Summarization Guided KV Cache Compression [21.816110519918876]
  Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing key-value ( KV) caches.<n>We present SGD-KV, a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation.<n>Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.
  arXiv  Detail & Related papers  (2026-09-03T00:31:11Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction [50.99402504483692]
  We propose a novel gating-based KV cache eviction method for frozen-weight language models.<n>Our approach integrates seamlessly into both the prefill and decoding stages.<n>Experiments show that our method maintains near-lossless performance while evicting up to 70% of the KV cache.
  arXiv  Detail & Related papers  (2026-01-25T03:07:54Z)
• Low-Rank Key Value Attention [3.7728602841318426]
  Transformer pretraining is increasingly constrained by memory and compute requirements.<n>We propose textitlow-rank KV adaptation (LRKV), a simple modification of multi-head attention.<n>LRKV is a drop-in replacement for standard multi-head attention.
  arXiv  Detail & Related papers  (2026-01-16T17:56:40Z)
• LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences [12.093166735658626]
  Key-Value (KV) cache succeeds in reducing redundant computations in auto-regressive models.<n>It introduces significant memory overhead, limiting its practical deployment in long-sequence scenarios.<n>Existing KV retrieval methods suffer from notable efficiency and accuracy bottlenecks due to per-token retrieval and coarse-grained page-level KV management.
  arXiv  Detail & Related papers  (2025-10-13T11:28:30Z)
• KV-Efficient VLA: A Method of Speed up Vision Language Model with RNN-Gated Chunked KV Cache [0.9238700679836854]
  Vision-Language-Action (VLA) models promise unified robotic perception and control, yet their scalability is constrained by the quadratic cost of attention and the unbounded growth of key-value (KV) memory during long-horizon inference.<n>We present KV-Efficient VLA, a model-agnostic memory compression framework that addresses these limitations by introducing a lightweight, training-friendly mechanism to selectively retain high-utility context.<n>Our method integrates seamlessly into existing autoregressive and hybrid VLA stacks, enabling scalable inference without modifying training pipelines or downstream control logic.
  arXiv  Detail & Related papers  (2025-09-20T02:04:24Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)
• MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache [17.58398289266989]
  Mini KV is a KV cache optimization method that simultaneously preserves long context task accuracy while significantly reducing KV cache size.<n>We show that Mini KV achieves 86% KV cache compression ratio while recovering over 98.5% of accuracy, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-27T06:10:49Z)
• Effectively Compress KV Heads for LLM [28.0801697946958]
  We propose a novel approach for compressing Key-Value ( KV) caches. Our method can compress half or even three-quarters of KV heads while maintaining performance comparable to the original LLMs.
  arXiv  Detail & Related papers  (2024-06-11T08:37:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.