Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction

• URL: http://arxiv.org/abs/2602.08585v1
• Date: Mon, 09 Feb 2026 12:23:38 GMT
• Title: Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction
• Abstract summary: Current KV cache eviction methods rely on instantaneous metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads.<n>In this paper, we propose that optimal budget allocation should be governed by the marginal utility in preserving long-term semantic information.<n>We implement a data-driven offline profiling protocol to facilitate the practical deployment of LU-KV.
• Score: 19.14455067106419
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference. Current KV cache eviction methods typically rely on instantaneous heuristic metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads. However, this overlooks the heterogeneity in predictive fidelity across attention heads. While certain heads prioritize the instantaneous contribution of tokens, others are dedicated to capturing long-horizon utility. In this paper, we propose that optimal budget allocation should be governed by the marginal utility in preserving long-term semantic information. Based on this insight, we propose LU-KV, a novel framework that optimizes head-level budget allocation through a convex-hull relaxation and a marginal-utility-based greedy solver to achieve near-optimal precision. Furthermore, we implement a data-driven offline profiling protocol to facilitate the practical deployment of LU-KV. Extensive evaluations on LongBench and RULER benchmarks demonstrate that LU-KV achieves an 80% reduction in KV cache size with minimal performance degradation, while simultaneously reducing inference latency and GPU memory footprint.

Related papers

• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM [12.14455026524814]
  Large language models (LLMs) excel at complex tasks like question answering and summarization.<n>Existing KV-cache eviction strategies leverage the sparsity in attention to selectively store a subset of tokens.<n>We introduce K-VEC, a novel coverage-aware KV-cache eviction strategy that prioritizes token coverage while evicting tokens in the cache.
  arXiv  Detail & Related papers  (2026-06-28T19:04:47Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache [11.676571773958145]
  omitting critical KV entries can induce substantial errors in model outputs.<n>Existing methods typically operate under fixed or adaptive token budgets.<n>We introduce Louver, a novel index structure tailored for efficient KV cache retrieval.
  arXiv  Detail & Related papers  (2026-05-07T17:37:56Z)
• DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing [30.82607063015274]
  This paper introduces DASH-KV, an innovative acceleration framework that reformulates attention as approximate nearest-neighbor search via asymmetric deep hashing.<n>Experiments on LongBench demonstrate that DASH-KV significantly outperforms state-of-the-art baseline methods while matching the performance of full attention, all while reducing inference complexity from O(N2) to linear O(N)
  arXiv  Detail & Related papers  (2026-04-21T11:33:24Z)
• Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction [50.99402504483692]
  We propose a novel gating-based KV cache eviction method for frozen-weight language models.<n>Our approach integrates seamlessly into both the prefill and decoding stages.<n>Experiments show that our method maintains near-lossless performance while evicting up to 70% of the KV cache.
  arXiv  Detail & Related papers  (2026-01-25T03:07:54Z)
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition [24.262712463465665]
  CurDKV is a novel, value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition.<n>Our approach approximates the dominant subspace of the attention output $softmax(QKT)V$, ensuring that the retained tokens best preserve the model's predictive behavior.
  arXiv  Detail & Related papers  (2025-09-18T15:04:06Z)
• Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache [67.47789629197857]
  We propose a training-free framework that exploits the heterogeneous roles of transformer head dimensions.<n>By projecting the long-context-insensitive dimensions onto Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients.<n>We show that FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack.
  arXiv  Detail & Related papers  (2025-06-13T15:35:54Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• Activation-aware Probe-Query: Effective Key-Value Retrieval for Long-Context LLMs Inference [56.71209737306054]
  We propose textbfActQKV, a training-free, textbfActivation-aware approach that dynamically determines probe-textbfQuery and leverages it to retrieve the relevant textbfKV pairs for inference.<n>Experiments on the Long-Bench and $infty$ Benchmarks demonstrate its state-of-the-art performance with competitive inference quality and resource efficiency.
  arXiv  Detail & Related papers  (2025-02-19T08:50:44Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time [44.89402186438295]
  Large Language Models (LLMs) have ignited an innovative surge of AI applications, marking a new era of exciting possibilities equipped with extended context windows. However, hosting these models is cost-prohibitive mainly due to the extensive memory consumption of KV Cache involving long-context modeling. We propose NACL, a general framework for long-context KV cache eviction that achieves more optimal and efficient eviction in a single operation during the encoding phase.
  arXiv  Detail & Related papers  (2024-08-07T10:31:07Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference [37.94892570127548]
  Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache.<n>Recent efforts aim to reduce KV cache size by evicting vast non-critical cache elements during runtime.<n>We propose Ada-KV, the first head-wise adaptive budget allocation strategy.
  arXiv  Detail & Related papers  (2024-07-16T09:53:32Z)
• Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks [21.815661269986425]
  We propose a novel KV cache merging approach, called KVMerger, to achieve adaptive KV cache compression for long-context tasks. Our approach is inspired by the intriguing observation that key states exhibit high similarity at the token level within a single sequence. We conduct extensive experiments to demonstrate the effectiveness of KVMerger for long-context tasks under constrained memory budgets.
  arXiv  Detail & Related papers  (2024-07-11T12:50:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.