KVpop -- Key-Value Cache Compression with Predictive Online Pruning

• URL: http://arxiv.org/abs/2607.05061v2
• Date: Wed, 08 Jul 2026 15:11:54 GMT
• Title: KVpop -- Key-Value Cache Compression with Predictive Online Pruning
• Abstract summary: We introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision.<n>The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps.<n> KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines.
• Score: 22.664867330302517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.

Related papers

• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• G-KV: Decoding-Time KV Cache Eviction with Global Attention [57.47409249054187]
  Large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths.<n> KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning.<n>We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance.
  arXiv  Detail & Related papers  (2025-11-29T14:21:33Z)
• Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution [2.894551569099569]
  We introduce $textbfExpected Attention, a training-free compression method that estimates KV pairs importance by predicting how future queries will attend to them.<n>Our method operates seamlessly across both prefilling and decoding phases, consistently outperforming state-of-the-art baselines in both scenarios.<n>$textbfwe release KVPress, a comprehensive library to enable researchers to implement and benchmark KV cache compression methods.
  arXiv  Detail & Related papers  (2025-10-01T08:12:14Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs? [79.58770714228983]
  Language models handle increasingly long contexts for tasks such as book summarization.<n>This leads to growing memory costs for the key-value ( KV) cache.<n>Many prior works have proposed ways of discarding KVs from memory, but their approaches are tailored to favorable settings.<n>We propose the * KV footprint* as a unified metric, which accounts for both the amount of KV entries stored and their lifespan in memory.
  arXiv  Detail & Related papers  (2025-06-20T16:21:12Z)
• LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning [21.761205124793175]
  extended reasoning sequences introduce significant GPU memory overhead due to increased key-value (KV) cache.<n>Existing KV cache compression methods mitigate memory bottlenecks but struggle in long reasoning tasks.<n>We propose LazyEviction, an observation window-based lagged eviction framework retaining latent recurring tokens by prioritized eviction based on tokens' recurrence patterns.
  arXiv  Detail & Related papers  (2025-06-19T02:25:04Z)
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management [48.904743679691414]
  FlowKV is a novel multi-turn isolation mechanism for KV Cache management.<n>It preserves the accumulated compressed KV cache from past turns.<n>It prevents the re-compression of older context and thereby mitigating catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-05-21T10:20:46Z)
• No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization [31.806112535762367]
  Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models(LLMs)
  arXiv  Detail & Related papers  (2024-02-28T06:34:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.