TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy

• URL: http://arxiv.org/abs/2608.27128v2
• Date: Mon, 31 Aug 2026 03:59:21 GMT
• Title: TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
• Abstract summary: Long-context inference is bottlenecked by the memory footprint of the key-value ( KV) cache.<n>We introduce TwinKV, a training-free, attention-free redundancy signal that detects whether a token's key has a near-duplicate elsewhere in context.<n>We compose TwinKV with four recent eviction policies across LongBench, LooGLE, RULER, and a short-context MMLU-Pro no-harm control at compression ratios $0.3,0.5,0.7$.
• Score: 32.29092613421357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-context inference is bottlenecked by the memory footprint of the key-value (KV) cache, especially for small models under tight resource budgets. Existing KV cache eviction methods score tokens using the model's attention distribution or, in attention-free variants, each key's distance from a global reference point. Using a controlled leave-one-out probe, we find that attention magnitude is unrelated to a token's causal contribution to the answer (Spearman $ρ=-0.004$), challenging the premise behind dominant eviction methods. We introduce TwinKV, a training-free, attention-free redundancy signal that detects whether a token's key has a near-duplicate elsewhere in context. Rather than replacing existing policies, TwinKV acts as a composable repair pass: given a policy's fixed retained set, it identifies evicted tokens with no surviving duplicate (\emph{orphans}) and retained tokens whose information is duplicated elsewhere (\emph{redundant donors}), then swaps them while preserving the original budget and scoring rule. We compose TwinKV with four recent eviction policies across LongBench, LooGLE, RULER, and a short-context MMLU-Pro no-harm control at compression ratios ${0.3,0.5,0.7}$. On Qwen3-4B, TwinKV improves a majority of configurations for two policies, is near-even for a third, and helps only a minority for a fourth adaptive baseline already near a performance ceiling; gains across the three non-ceiling policies are smallest at the loosest ratio. On RULER with Llama-3.2-1B, however, that fourth policy improves in every evaluated cell because its Alone score leaves substantial room to improve. More broadly, Llama-3.2-1B shows a smaller average LongBench gain but a higher fraction of improved cells on LongBench and LooGLE than Qwen3-4B, plus a clean RULER win. We also identify few-shot classification exemplars as a task structure where TwinKV does not help on either model.

Related papers

• VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs [7.825137277264239]
  Key-value cache is main memory bottleneck in long-context large language model (LLM) inference.<n>VarRate is a training-free KV that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank.<n>VarRate is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.
  arXiv  Detail & Related papers  (2026-07-16T23:03:14Z)
• MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference [12.316173390280609]
  Autoregressive decoding in Transformer-based language models relies on the KV cache.<n> KV cache eviction addresses this by retaining a fixed-size subset of key-value pairs and discarding the rest.<n>We propose MomentKV, which maintains compact, small-size moment statistics over the evicted token set.
  arXiv  Detail & Related papers  (2026-06-01T02:08:40Z)
• GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs [97.36238579001544]
  Large language models (LLMs) with extended context lengths rely on the key-value ( KV) cache to support attention over prior tokens.<n>Maintaining the KV cache incurs substantial memory overhead, motivating KV-cache compression methods.<n>We propose GRKV, a training-free KV-cache merging method that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs.
  arXiv  Detail & Related papers  (2026-05-29T10:16:30Z)
• NestedKV: Nested Memory Routing for Long-Context KV Cache Compression [35.62789874560166]
  NestedKV is a key-only KV cache compression method inspired by the Continuum Memory System in Nested Learning.<n>It scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner.
  arXiv  Detail & Related papers  (2026-05-26T08:14:39Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• KVSculpt: KV Cache Compression as Distillation [7.085426079187912]
  KV cache compression is critical for efficient long-context LLM inference.<n>Existing methods range from pure eviction -- selecting which KV pairs to keep -- to merging, which combines similar pairs into fewer ones.<n>We propose KVSculpt, which moves to the other end of this spectrum.<n>Keys are optimized via L-BFGS and values are solved in closed form via least squares.
  arXiv  Detail & Related papers  (2026-03-29T19:14:25Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models [110.06476624089679]
  We introduce a novel approach for implementing the KV cache which significantly reduces its memory footprint. Our approach is based on the observation that a small portion of tokens contributes most of the value when computing attention scores. We propose Heavy Hitter (H$$O), a KV cache eviction policy that dynamically retains a balance of recent and H$$ tokens.
  arXiv  Detail & Related papers  (2023-06-24T20:11:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.