Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective

• URL: http://arxiv.org/abs/2604.25975v1
• Date: Tue, 28 Apr 2026 12:28:04 GMT
• Title: Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
• Abstract summary: Key-value (KV) caching is essential for large language model inference, yet it poses a critical bottleneck for long-context generation.<n>This work rethinks KV cache eviction through the lens of the Information Bottleneck principle.<n>We introduce CapKV, a capacity-aware eviction method that directly targets information preservation via a log-determinant approximation.
• Score: 26.25624771395719
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Key-value (KV) caching is essential for large language model inference, yet its memory overhead poses a critical bottleneck for long-context generation. Existing eviction policies predominantly rely on empirical heuristics, lacking a rigorous theoretical foundation. This work rethinks KV cache eviction through the lens of the Information Bottleneck principle. Under a linear-Gaussian surrogate of attention, we derive a closed-form mutual information objective that characterizes the effective information capacity of a retained KV cache subset. This formulation reveals that a wide range of existing eviction strategies can be interpreted as different approximations of the same capacity-maximization principle. Guided by this insight, we introduce CapKV, a capacity-aware eviction method that directly targets information preservation via a log-determinant approximation using statistical leverage scores. This approach replaces heuristic selection with a theoretically grounded mechanism that preserves the maximum predictive signal. Extensive experiments across multiple models and long-context benchmarks show that CapKV consistently outperforms prior methods, achieving a better trade-off between memory efficiency and generational fidelity.

Related papers

• Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation [26.316211624524197]
  Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models.<n>We introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection.
  arXiv  Detail & Related papers  (2026-09-08T02:08:57Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• Reformulating KV Cache Eviction Problem for Long-Context LLM Inference [2.0590127484132523]
  Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value ( KV) Cache growth.<n>Existing KV Cache eviction methods rely on local attention weights, neglecting the influence of value representations, output projection, and inter-head interactions.<n>We introduce LaProx, a novel eviction strategy that explicitly models the multiplicative interaction between attention maps and projected value states.
  arXiv  Detail & Related papers  (2026-05-08T04:37:22Z)
• Learning to Evict from Key-Value Cache [17.365511268829703]
  We introduce KV Policy, a framework for learning to rank tokens by their predicted usefulness for future decoding.<n> evaluated across two different model families on the long-context benchmark RULER and the multi-turn dialogue benchmark OASST2-4k.<n>Results demonstrate that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management.
  arXiv  Detail & Related papers  (2026-02-10T19:34:15Z)
• Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction [19.14455067106419]
  Current KV cache eviction methods rely on instantaneous metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads.<n>In this paper, we propose that optimal budget allocation should be governed by the marginal utility in preserving long-term semantic information.<n>We implement a data-driven offline profiling protocol to facilitate the practical deployment of LU-KV.
  arXiv  Detail & Related papers  (2026-02-09T12:23:38Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition [24.262712463465665]
  CurDKV is a novel, value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition.<n>Our approach approximates the dominant subspace of the attention output $softmax(QKT)V$, ensuring that the retained tokens best preserve the model's predictive behavior.
  arXiv  Detail & Related papers  (2025-09-18T15:04:06Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs [0.0]
  Key-Value ( KV) cache quantization has become a widely adopted optimization technique for large language models (LLMs) inference.<n>Recent studies have emphasized the importance of preserving the original precision of KVs for the first few tokens to ensure the protection of attention sinks.<n>We introduce textittextbf KVSink, a plug-and-play method that effectively predicts sink tokens with negligible overhead.
  arXiv  Detail & Related papers  (2025-08-06T09:40:09Z)
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management [48.904743679691414]
  FlowKV is a novel multi-turn isolation mechanism for KV Cache management.<n>It preserves the accumulated compressed KV cache from past turns.<n>It prevents the re-compression of older context and thereby mitigating catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-05-21T10:20:46Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models [28.244034916473804]
  Generative inference in Large Language Models (LLMs) is impeded by the growing memory demands of Key-Value (KV) cache.<n>Traditional KV cache eviction strategies discard less critical KV pairs based on attention scores, leading to issues such as context loss or hallucinations.<n>We introduce Dynamic Discriminative Operations (D2O), a KV cache compression method that optimize KV cache size dynamically and discriminatively at two levels without fine-tuning.
  arXiv  Detail & Related papers  (2024-06-18T20:01:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.