G-KV: Decoding-Time KV Cache Eviction with Global Attention

• URL: http://arxiv.org/abs/2512.00504v1
• Date: Sat, 29 Nov 2025 14:21:33 GMT
• Title: G-KV: Decoding-Time KV Cache Eviction with Global Attention
• Abstract summary: Large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths.<n> KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning.<n>We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance.
• Score: 57.47409249054187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on: https://github.com/microsoft/G-KV.

Related papers

• GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs [97.36238579001544]
  Large language models (LLMs) with extended context lengths rely on the key-value ( KV) cache to support attention over prior tokens.<n>Maintaining the KV cache incurs substantial memory overhead, motivating KV-cache compression methods.<n>We propose GRKV, a training-free KV-cache merging method that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs.
  arXiv  Detail & Related papers  (2026-05-29T10:16:30Z)
• Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility [31.124813359673073]
  We introduce Self-Pruned Key-Value Attention (SP-KV), a mechanism designed to predict future KV utility.<n>A lightweight utility predictor scores each key-value pair, and while recent KVs are always available via a local window, older pairs are written in the cache.<n>The mechanism adapts to the input and typically reduces the KV cache size by a factor of $3$ to $10times$.
  arXiv  Detail & Related papers  (2026-05-13T18:58:16Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression [18.12657364501536]
  FAEDKV is a novel, training-free KV cache compression framework.<n>It preserves both early and recent contextual information.<n>Experiments on LongBench benchmark demonstrate FAEDKV's superiority over existing methods by up to 22%.
  arXiv  Detail & Related papers  (2025-07-26T18:20:25Z)
• AttentionPredictor: Temporal Patterns Matter for KV Cache Compression [64.75459635661562]
  We propose AttentionPredictor, which is the first learning-based method to directly predict attention patterns for KV cache compression and critical token identification.<n> AttentionPredictor accurately predicts the attention score and shares the unified prediction model, which consumes negligible memory.<n>By retaining most of the attention information, AttentionPredictor achieves 13$times$ KV cache compression and 5.6$times$ speedup in a cache offloading scenario.
  arXiv  Detail & Related papers  (2025-02-06T13:41:46Z)
• ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression [10.003118268356017]
  Long context poses significant challenges for inference efficiency.<n>We introduce ClusterKV, which recalls tokens at the granularity of semantic clusters.<n>Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths.
  arXiv  Detail & Related papers  (2024-12-04T10:58:27Z)
• Eigen Attention: Attention in Low-Rank Space for KV Cache Compression [9.080678336379528]
  We propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. We show that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance.
  arXiv  Detail & Related papers  (2024-08-10T22:47:12Z)
• A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression [13.981807478365452]
  Existing approaches to reduce the Key-Value cache size involve either fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce the sequence length. We find a clear correlation between the $L$ and the attention scores over cached KV pairs, where a low $L$ of a key embedding leads to a high attention score during decoding. Our experimental results show that this simple strategy can reduce the KV cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy.
  arXiv  Detail & Related papers  (2024-06-17T11:35:16Z)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference [78.65321721142624]
  We focus on a memory bottleneck imposed by the key-value ( KV) cache. Existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs. We propose LESS, a simple integration of a constant sized cache with eviction-based cache methods.
  arXiv  Detail & Related papers  (2024-02-14T18:54:56Z)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [82.08922896531618]
  We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) We conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens.
  arXiv  Detail & Related papers  (2023-10-03T05:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.