When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression

• URL: http://arxiv.org/abs/2605.08234v1
• Date: Thu, 07 May 2026 00:36:59 GMT
• Title: When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
• Abstract summary: Long-context LLM inference is bottlenecked by the memory and bandwidth cost of reading large KV caches during decoding.<n> KV compression reduces this cost by keeping only part of the cache, but task accuracy alone does not identify why a selector succeeds or fails.<n>A selector can fail at three steps: it may miss the evidence future decoding needs, give high scores to tokens that do not affect the output, or break related evidence when fitting scores into a small cache.<n>We introduce a fixed-contract diagnostic that holds the selector's setup fixed and changes one decision slot at a time.
• Score: 12.959497604836956
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context LLM inference is bottlenecked by the memory and bandwidth cost of reading large KV caches during decoding. KV compression reduces this cost by keeping only part of the cache, but task accuracy alone does not identify why a selector succeeds or fails. A selector can fail at three steps: it may miss the evidence future decoding needs, give high scores to tokens that do not affect the output, or break related evidence when fitting scores into a small cache. We introduce a fixed-contract diagnostic that holds the selector's setup fixed and changes one decision slot at a time. For value ranking, the probe combines a block's attention mass with the estimated output change from removing it. On LongBench across three models and two budgets, the probe is positive on 72.6% of positive-margin cells and 32.4% of nonpositive-margin cells. NeedleBench M-RT at 32k and a RULER 8k check probe support closure under branched retrieval, and a 264-cell sign evaluation separates support recovery and output-value ranking from leverage effects near the boundary. The resulting order is to recover decode-side evidence, rank its output value, and preserve coupled evidence during projection.

Related papers

• Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning [78.75986976062454]
  Large language models achieve superior performance on tasks that require extended reasoning.<n>Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones.<n>We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all.
  arXiv  Detail & Related papers  (2026-09-03T06:38:38Z)
• D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory [10.177699115336425]
  D$2$ACCI protocol promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability.<n>We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2.
  arXiv  Detail & Related papers  (2026-08-18T13:18:38Z)
• KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models [5.349156411852564]
  KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why.<n>We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions.
  arXiv  Detail & Related papers  (2026-08-10T10:38:05Z)
• Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions [87.95757610455173]
  We introduce an injectively aligned leave-one-cell-out protocol over support x operation grids and SO-OPF.<n>We separate two questions that aggregate scores conflate: whether the carrier composes held-out bindings when the grid is known, and whether that grid can be recovered from flat cell labels.
  arXiv  Detail & Related papers  (2026-08-06T15:38:55Z)
• Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression [42.94199567901975]
  KV cache compression is commonly evaluated by final-answer accuracy, implicitly assuming that preserving the answer also preserves the reasoning that supports it.<n>We test this assumption for large reasoning models and show that it can fail.<n>We evaluate ten token-eviction KV compression methods and one quantization method on three models across mathematical reasoning, scientific QA, clinical calculation, and long-context retrieval.
  arXiv  Detail & Related papers  (2026-08-03T03:03:53Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization [20.158719055939383]
  KV-cache quantization is validated today by offline benchmark averages.<n>A deployed system cannot tell whether compression is damaging the request it is serving right now.<n>We give it a provably sound runtime meter -- a "DTrace for KV quantization"
  arXiv  Detail & Related papers  (2026-07-30T11:04:45Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning [0.16921396880325776]
  We introduce Support Vector Attention (SV-Attention), a memory whose weights are support coefficients of a one-class Sfp with fixed box parameter C.<n>Its active-set partition gives reserve tokens exactly zero weight, output weights; a incremental incremental solver token to recover the state produced by retraining without it under the same C.<n>We also demonstrate surgical eviction, exact editing, patient-record deletion, and a forgettable retrieval memory over real sentence embeddings.
  arXiv  Detail & Related papers  (2026-07-13T23:13:57Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory [4.425240080125479]
  We present a scale-conditioned evaluation protocol for agent memory under evidence-preserving growth.<n>For each query, task evidence is held fixed while irrelevant sessions are added.<n>The protocol logs agent--memory trajectories and reports four diagnostics.
  arXiv  Detail & Related papers  (2026-05-08T06:22:31Z)
• Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators [45.88028371034407]
  KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA)<n>We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA)
  arXiv  Detail & Related papers  (2026-05-07T22:26:27Z)
• EviMem: Evidence-Gap-Driven Iterative Retrieval for Long-Term Conversational Memory [19.027571478872]
  We present EviMem, combining IRIS (Iterative Retrieval via Insufficiency Signals) and LaceMem (Layered Architecture for Conversational Evidence Memory)<n>On LoCoMo, EviMem improves Judge Accuracy over MIRIX on temporal (73.3% to 81.6%) and multi-hop (65.9% to 85.2%) questions at 4.5x lower latency.
  arXiv  Detail & Related papers  (2026-04-30T10:37:04Z)
• WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference [56.297697169678095]
  WISV (Wireless-Informed Semantic Verification) is a novel distributed speculative decoding framework.<n>We show that WISV achieves up to a 60.8% increase in accepted length, a 37.3% reduction in interaction rounds, and a 31.4% improvement in end-to-end latency.<n>We validate WISV on a hardware testbed comprising an NVIDIA Jetson AGX Orin and an A40-equipped server.
  arXiv  Detail & Related papers  (2026-04-20T01:29:56Z)
• Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference [54.467557491325046]
  We propose Pre-hoc Sparsity (PrHS), which selects KV entries before attention scoring and provides explicit accuracy control.<n>PrHS reduces retrieval overhead by over 90%, achieving 3x higher retrieval sparsity than HShare at matched or better accuracy.<n>It incurs under 1% average degradation on LongBench, lowers attention FLOPs by about 15% versus prior baselines, and yields a 9.9x speedup in attention-operator latency and 2.8x higher throughput.
  arXiv  Detail & Related papers  (2026-02-09T07:05:23Z)
• S$^3$-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference [11.779449360037518]
  We present S3-Attention, a memory-first inference-time framework that treats long-context processing as attention-aligned endogenous retrieval.<n>S3-Attention decodes transient key and query projections into top-k sparse feature identifiers using lightweight sparse autoencoders.<n>It constructs a CPU-based inverted index mapping features to token positions or spans during a single streaming scan.
  arXiv  Detail & Related papers  (2026-01-25T05:25:22Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.