QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

• URL: http://arxiv.org/abs/2608.05326v1
• Date: Wed, 05 Aug 2026 18:29:47 GMT
• Title: QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
• Abstract summary: We show that token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states.<n>We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction.
• Score: 13.920782070418246
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention

Related papers

• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference [12.316173390280609]
  Autoregressive decoding in Transformer-based language models relies on the KV cache.<n> KV cache eviction addresses this by retaining a fixed-size subset of key-value pairs and discarding the rest.<n>We propose MomentKV, which maintains compact, small-size moment statistics over the evicted token set.
  arXiv  Detail & Related papers  (2026-06-01T02:08:40Z)
• IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference [33.37463192332294]
  We introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens.<n>We propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state.<n>Our method enables accurate long-context inference under a bounded KV budget.
  arXiv  Detail & Related papers  (2026-05-25T06:29:43Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• FASA: Frequency-aware Sparse Attention [56.26881872333624]
  We propose FASA, a novel framework that achieves query-aware token eviction by dynamically predicting token importance.<n>Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full attention head.<n>Across a spectrum of long-context tasks, FASA consistently outperforms all token-eviction baselines and achieves near-oracle accuracy.
  arXiv  Detail & Related papers  (2026-02-03T06:09:06Z)
• G-KV: Decoding-Time KV Cache Eviction with Global Attention [57.47409249054187]
  Large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths.<n> KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning.<n>We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance.
  arXiv  Detail & Related papers  (2025-11-29T14:21:33Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning [21.761205124793175]
  extended reasoning sequences introduce significant GPU memory overhead due to increased key-value (KV) cache.<n>Existing KV cache compression methods mitigate memory bottlenecks but struggle in long reasoning tasks.<n>We propose LazyEviction, an observation window-based lagged eviction framework retaining latent recurring tokens by prioritized eviction based on tokens' recurrence patterns.
  arXiv  Detail & Related papers  (2025-06-19T02:25:04Z)
• CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction [6.0863739814718025]
  Token eviction is a widely adopted post-training methodology designed to alleviate the bottlenecks by evicting less important tokens from the cache.<n>We propose a simple eviction criterion based on the contribution of cached tokens to attention outputs.<n>We show that CAOTE, when combined with the state-of-the-art attention score-based methods, always improves accuracies on the downstream task.
  arXiv  Detail & Related papers  (2025-04-18T19:46:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.