Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation

• URL: http://arxiv.org/abs/2609.08131v1
• Date: Tue, 08 Sep 2026 02:08:57 GMT
• Title: Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
• Abstract summary: Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models.<n>We introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection.
• Score: 26.316211624524197
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models. However, existing policies predominantly rely on empirical heuristics, lacking a rigorous characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we rethink KV cache eviction through the lens of local information geometry, modeling the attention process as a nonlinear Gaussian communication channel. By performing a first-order Taylor expansion of the attention mapping, we derive the Jacobian Information Capacity, a novel objective that explicitly captures query relevance, softmax sensitivity, and structural diversity. Guided by this theory, we introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection. Extensive experiments across diverse architectures and benchmarks demonstrate that \textsc{Jacap} delivers superior performance in most scenarios, particularly in high-compression regimes.

Related papers

• Sparse Attention for Dense Open-Vocabulary Prediction in CLIP [54.45759517856271]
  We study an inference-time substitution of the row-wise softmax in the final visual self-attention layers.<n>Because entmax applies a data-dependent threshold that maps low scores exactly to zero, it acts as an implicit denoiser.<n>We evaluate on open-vocabulary tasks-dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD)
  arXiv  Detail & Related papers  (2026-07-08T08:30:48Z)
• Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM [12.14455026524814]
  Large language models (LLMs) excel at complex tasks like question answering and summarization.<n>Existing KV-cache eviction strategies leverage the sparsity in attention to selectively store a subset of tokens.<n>We introduce K-VEC, a novel coverage-aware KV-cache eviction strategy that prioritizes token coverage while evicting tokens in the cache.
  arXiv  Detail & Related papers  (2026-06-28T19:04:47Z)
• Information-Aware KV Cache Compression for Long Reasoning [71.77807044404499]
  Existing KV cache compression methods mainly rely on attention weights to estimate token importance.<n>We propose textbfInfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals.
  arXiv  Detail & Related papers  (2026-06-25T11:03:37Z)
• Reformulating KV Cache Eviction Problem for Long-Context LLM Inference [2.0590127484132523]
  Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value ( KV) Cache growth.<n>Existing KV Cache eviction methods rely on local attention weights, neglecting the influence of value representations, output projection, and inter-head interactions.<n>We introduce LaProx, a novel eviction strategy that explicitly models the multiplicative interaction between attention maps and projected value states.
  arXiv  Detail & Related papers  (2026-05-08T04:37:22Z)
• Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective [26.25624771395719]
  Key-value (KV) caching is essential for large language model inference, yet it poses a critical bottleneck for long-context generation.<n>This work rethinks KV cache eviction through the lens of the Information Bottleneck principle.<n>We introduce CapKV, a capacity-aware eviction method that directly targets information preservation via a log-determinant approximation.
  arXiv  Detail & Related papers  (2026-04-28T12:28:04Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• Transformers Learn Faster with Semantic Focus [57.97235825738412]
  We study sparse transformers in terms of learnability and generalization.<n>We find that input-dependent sparse attention models appear to converge faster and generalize better than standard attention models.
  arXiv  Detail & Related papers  (2025-06-17T01:19:28Z)
• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• Anchor Attention, Small Cache: Code Generation with Large Language Models [15.94784908771546]
  Current practices in NLP often use sparse attention which may, unfortunately, lead to substantial inaccuracies, or hallucinations, in code generation tasks. We propose a novel approach, AnchorCoder, which features token-wise anchor attention designed to extract and compress contextual information. It can consistently achieve a significant (at least 70%) reduction in KV cache requirements, while preserving the majority of model's performance.
  arXiv  Detail & Related papers  (2024-11-11T02:47:05Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference [37.94892570127548]
  Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache.<n>Recent efforts aim to reduce KV cache size by evicting vast non-critical cache elements during runtime.<n>We propose Ada-KV, the first head-wise adaptive budget allocation strategy.
  arXiv  Detail & Related papers  (2024-07-16T09:53:32Z)
• QAQ: Quality Adaptive Quantization for LLM KV Cache [3.163526369095745]
  A bottleneck in model deployment emerges due to the linear expansion of the Key-Value cache with the context length. We propose QAQ, a Quality Adaptive Quantization scheme for the KV cache.
  arXiv  Detail & Related papers  (2024-03-07T16:42:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.