CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

• URL: http://arxiv.org/abs/2605.24786v1
• Date: Sun, 24 May 2026 00:15:34 GMT
• Title: CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
• Abstract summary: We introduce CONF-KV, a KV-cache manager that chooses the per-step cache budget.<n> tokens are ranked by a composite of accumulated attention mass and recency.<n>On 75 VisualWebArena tasks it retains 95.3% of full- KV success at 2.8 times lower peak memory.
• Score: 8.82843279804122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon LLM inference turns the key--value (KV) cache into the dominant GPU memory consumer and makes per-token attention increasingly expensive. Many common eviction policies use static recency windows or historical attention, leaving unused a signal computed on every decoding step: the model's current uncertainty. We introduce CONF-KV, a KV-cache manager that converts the next-token distribution into a scalar confidence score and uses it to choose the per-step cache budget, retaining more context when the model is uncertain and pruning aggressively when it is confident. Within each budget, tokens are ranked by a composite of accumulated attention mass and recency, while a protected recent window preserves local coherence. We combine the policy with blockwise online-softmax attention, mixed FP16/INT8 storage, and a pyramidal per-layer budget variant. Across four model families and generated lengths up to 4K, CONF-KV stays near the footprint of a fixed 512-token sliding window while remaining within 1.5--2.1 perplexity points of full KV. On Needle-in-a-Haystack up to 32K tokens, CONF-KV reaches 91.4% retrieval accuracy versus 53.8% for sliding windows and 80.6% for H2O; on 75 VisualWebArena tasks it retains 95.3% of full-KV success at 2.8 times lower peak memory.

Related papers

• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention [0.0]
  Minima-KV is a retention-preserving hierarchy for mixed-format paged attention.<n>Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3.<n> format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax state.
  arXiv  Detail & Related papers  (2026-08-24T21:21:58Z)
• Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse [3.9548951874725433]
  We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill.<n>We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension.<n>This linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply.
  arXiv  Detail & Related papers  (2026-08-04T16:26:47Z)
• FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference [0.0]
  FreqDepthKV is an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals.<n>A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits.<n>Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets.
  arXiv  Detail & Related papers  (2026-07-07T17:26:28Z)
• IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference [12.04455190856202]
  Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.<n>We introduce IntentKV, learned KV pruning that keeps the base LLM frozen.<n>IntentKV matches the no-pruning full-cache baseline with almost no accuracy drop under tight KV budgets.
  arXiv  Detail & Related papers  (2026-06-06T15:54:48Z)
• TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference [11.997332729897899]
  Key-value ( KV) caching is critical for efficient inference in large language models (LLMs)<n>We propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache.<n> experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.
  arXiv  Detail & Related papers  (2026-03-27T09:01:23Z)
• Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression [16.495900730787955]
  Key-value ( KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length.<n>We propose MixedDimKV, a mixed-dimension KV cache compression method that allocates dimensions to tokens at a more granular level.<n>Our solution maintains 100% accuracy at a 50K context length while using as little as 0.26% of the cache.
  arXiv  Detail & Related papers  (2026-03-21T03:21:43Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference [6.864810630905683]
  Existing defenses, such as per-user isolation, eliminate leakage but degrade performance by up to 38.9% in time-to-first-token (TTFT)<n>We introduce SafeKV, a privacy-aware KV-cache management framework that selectively shares non-sensitive entries while confining sensitive content to private caches.<n>Our evaluation shows that SafeKV mitigates 94% - 97% of timing-based side-channel attacks.
  arXiv  Detail & Related papers  (2025-08-11T19:55:44Z)
• HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs [13.013668526921778]
  Existing KV cache compression methods exhibit noticeable performance degradation when memory is reduced by more than 85%.<n>We propose HCAttention, a heterogeneous attention framework that integrates key quantization, value offloading, and dynamic KV eviction.<n> Experimental results on the LongBench benchmark demonstrate that our approach preserves the accuracy of full-attention model while shrinking the KV cache memory footprint to 25% of its original size.
  arXiv  Detail & Related papers  (2025-07-26T06:43:14Z)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models [77.84539432982307]
  We propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV)<n>R-KV preserves nearly 100% of the full KV cache performance using only 10% of the KV cache.<n>Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache.
  arXiv  Detail & Related papers  (2025-05-30T02:03:24Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing [58.29726147780976]
  We propose a plug-and-play method called textit KVSharer, which shares the KV cache between layers to achieve layer-wise compression. Experiments show that textit KVSharer can reduce KV cache computation by 30%, thereby lowering memory consumption. We verify that textit KVSharer is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.
  arXiv  Detail & Related papers  (2024-10-24T08:06:41Z)
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling [38.732413451399]
  Pyramid KV is a novel and effective KV cache compression method.<n>We show that Pyramid KV matches the performance of models with a full KV cache while retaining only 12% of the KV cache.<n>In the Needle-in-a-Haystack experiment, Pyramid KV outperforms competing methods in maintaining long-context comprehension.
  arXiv  Detail & Related papers  (2024-06-04T07:51:30Z)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget [29.208289711639853]
  optimizing the Key-Value cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms treat all layers equally, allocating the same KV budget to each layer. We find that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions.
  arXiv  Detail & Related papers  (2024-04-07T03:08:14Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.