Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression

• URL: http://arxiv.org/abs/2608.01631v1
• Date: Mon, 03 Aug 2026 03:03:53 GMT
• Title: Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
• Abstract summary: KV cache compression is commonly evaluated by final-answer accuracy, implicitly assuming that preserving the answer also preserves the reasoning that supports it.<n>We test this assumption for large reasoning models and show that it can fail.<n>We evaluate ten token-eviction KV compression methods and one quantization method on three models across mathematical reasoning, scientific QA, clinical calculation, and long-context retrieval.
• Score: 42.94199567901975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: KV cache compression is commonly evaluated by final-answer accuracy, implicitly assuming that preserving the answer also preserves the reasoning that supports it. We test this assumption for large reasoning models and show that it can fail: under compression, correct answers and the validity of their visible supporting rationales can be preserved at different rates. We study this failure with a controlled fixed-trace replay protocol, which holds reasoning content fixed and isolates whether compression preserves usable information from an already available trace. We evaluate ten token-eviction KV compression methods and one quantization method on three models across mathematical reasoning, scientific QA, clinical calculation, and long-context retrieval. We measure final accuracy, answer-chain consistency, and perturbation faithfulness. Across tasks, token-eviction methods can preserve competitive final-answer accuracy while substantially degrading chain support or perturbation faithfulness. We call this the answer-evidence gap. A coverage-preserving quantization control is substantially less affected, suggesting that the failure is tied less to KV memory reduction itself than to losing access to parts of the reasoning trace. Code is available at https://github.com/famous-blue-raincoat/Safe_KV_Compress.

Related papers

• A Probabilistic Interpretation of KV Cache Eviction [52.196873305708955]
  Cache eviction can achieve higher throughput by evicting some entries from the KV cache.<n>This paper formalizes the problem of KV eviction through the lens of probabilistic reasoning.<n>We show that this probabilistic version of KV eviction coupled with decode time correction is more robust to different tasks.
  arXiv  Detail & Related papers  (2026-08-28T12:57:07Z)
• Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought [13.898770874357446]
  Chain-of-Thought prompting incurs substantial computational costs due to over-reasoning.<n>We propose CAID, a training-free metric grounded in information theory that identifies low-utility steps.
  arXiv  Detail & Related papers  (2026-07-13T08:50:26Z)
• CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning [22.944748148277146]
  CopT is a reformulated reasoning pipeline that reverses the usual order of thinking and answering.<n>CopT elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer.<n>CopT improves peak accuracy by up to 23% and reduces token usage by up to 57% at comparable or higher accuracy.
  arXiv  Detail & Related papers  (2026-05-19T16:28:53Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding? [19.77569541429818]
  Speculative decoding accelerates LLM inference, but SOTA hidden-state-based drafters suffer from long-range decay.<n>We revisit long-range decay from the perspective of context information preservation.<n>We introduce KVShot, a diagnostic framework that compares three reuse paradigms.
  arXiv  Detail & Related papers  (2026-04-29T08:25:01Z)
• Shorter, but Still Trustworthy? An Empirical Study of Chain-of-Thought Compression [19.669117846064562]
  Long chain-of-thought (Long-CoT) reasoning models have motivated a growing body of work on compressing reasoning traces to reduce inference cost.<n>We conduct the first systematic empirical study of how CoT compression affects model trustworthiness.<n>We find that CoT compression frequently introduces trustworthiness regressions and that different methods exhibit markedly different degradation profiles across dimensions.
  arXiv  Detail & Related papers  (2026-04-05T13:43:12Z)
• Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck [12.360124156284305]
  Existing "Budget Forcing" methods suppress both essential reasoning and redundant filler.<n>We recast efficient reasoning as a lossy compression problem under the Information Bottleneck (IB) principle.<n>In contrast to naive token-counting-based approaches, we introduce a semantic prior that measures token cost by surprisal under a language model prior.
  arXiv  Detail & Related papers  (2026-03-09T14:56:57Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)
• Towards Evading the Limits of Randomized Smoothing: A Theoretical Analysis [74.85187027051879]
  We show that it is possible to approximate the optimal certificate with arbitrary precision, by probing the decision boundary with several noise distributions. This result fosters further research on classifier-specific certification and demonstrates that randomized smoothing is still worth investigating.
  arXiv  Detail & Related papers  (2022-06-03T17:48:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.