KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models

• URL: http://arxiv.org/abs/2608.09412v1
• Date: Mon, 10 Aug 2026 10:38:05 GMT
• Title: KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
• Abstract summary: KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why.<n>We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions.
• Score: 5.349156411852564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions. First, a 25-method taxonomy groups methods into five mechanism families and links them to eight verified implementations and their valid diagnostic measurements. Second, for every supported method setting, we evaluate all sources in each fixed split against a per-source FullCache control before selecting FullCache-correct/compressed-wrong (C-to-W) rows separately for each method-setting, so no compressor defines another's test set. Third, a common record format links paired outputs and run metadata to cache, likelihood, attention, and decoding measurements with explicit applicability states. On Qwen3-8B, four evidence-aware workloads yield 59 800 supported compressed runs over 2600 sources and 12 520 C-to-W rows. Under fixed diagnostic rules, 63.2% have low or partial measured/projected coverage. Only 19 rows (0.2%) combine high measured/projected coverage with strong likelihood drift; another 2,126 (17.0%) preserve structural position addressability, for which representation fidelity remains unknown, while showing the same drift. Against C-to-C success controls, all ten diagnostics separate failed from successful compression (stratified AUROC 0.684-0.871). Among 96 reproducible low-EAR failures, a controlled 4x evidence-attention boost repairs 29.2%, versus 6.3% under a count-matched sham intervention and 3.3% degradation on matched C-to-C controls. Code and data are available at https://github.com/ChosenQC/KVDiagnosis.

Related papers

• Data Diversity, Not Frequency Invariance: A Controlled and Self-Audited Study of Compression-Robust Deepfake Detection [0.26856688022781555]
  Frequency features and compression-invariant representation learning are widely assumed to be key to deepfake detection that survives video compression.<n>We test this with CAFRL - block-DCT and FFT-phase streams, compression-level-conditioned band attention, and adversarial (gradient-reversal) compression.<n>Under a pre-registered protocol with capacity- and augmentation-matched controls, a plain EfficientNet-B0 on multi-quality data beat CAFRL as specified at every compression level.
  arXiv  Detail & Related papers  (2026-08-26T12:05:10Z)
• D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory [10.177699115336425]
  D$2$ACCI protocol promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability.<n>We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2.
  arXiv  Detail & Related papers  (2026-08-18T13:18:38Z)
• Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression [42.94199567901975]
  KV cache compression is commonly evaluated by final-answer accuracy, implicitly assuming that preserving the answer also preserves the reasoning that supports it.<n>We test this assumption for large reasoning models and show that it can fail.<n>We evaluate ten token-eviction KV compression methods and one quantization method on three models across mathematical reasoning, scientific QA, clinical calculation, and long-context retrieval.
  arXiv  Detail & Related papers  (2026-08-03T03:03:53Z)
• Control Under Compression: Reliability Frontiers for Tool-Using Agents [0.0]
  We introduce CompressAgent, an environment-verified benchmark for ACC compression.<n>At 75% retained context, generic rewriting and section-based compression achieve 92.7% and 92.4% success.<n>At retained-context budgets from 25% to 10%, executable protocols become fragile.
  arXiv  Detail & Related papers  (2026-08-02T07:43:44Z)
• EduPluginBench: Executable Assurance for AI-Generated Educational Plugins [0.0]
  We introduce negative provenance, an executable benchmark and staged admission method for generated plugins in software ecosystems.<n>A frozen transfer study of 600 unmodified generations from two current coding models found that 300/600 parsed, but none passed P0 or achieved P0-P4 conformance.<n>An earlier 540-generation diagnostic found that post-hoc bounded repair yielded 112 P0 passes, all nonconforming, with recall increasing from 13.4% to 100%.
  arXiv  Detail & Related papers  (2026-08-01T16:22:35Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• How Query Visibility Changes KV-Cache Compression Rankings: A Matched-Budget Audit [5.366718741784969]
  KV-cache compression methods are predominantly evaluated with the query appended to the context before compression.<n>We present a matched-budget audit of six published compression methods against three trivial baselines.
  arXiv  Detail & Related papers  (2026-07-11T09:30:35Z)
• From CVE to CWE: Syscall-Based HIDS Generalisation [41.99844472131922]
  In operational settings, defenders need to recognise new exploits of an already known type of weakness.<n>We empirically examine whether a one-class anomaly detector trained on the normal behaviour of a set of CVEs that share a Common Weaknession (CWE) class generalises to a different, unseen CVE inside the same class.
  arXiv  Detail & Related papers  (2026-06-21T16:34:39Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression [12.959497604836956]
  Long-context LLM inference is bottlenecked by the memory and bandwidth cost of reading large KV caches during decoding.<n> KV compression reduces this cost by keeping only part of the cache, but task accuracy alone does not identify why a selector succeeds or fails.<n>A selector can fail at three steps: it may miss the evidence future decoding needs, give high scores to tokens that do not affect the output, or break related evidence when fitting scores into a small cache.<n>We introduce a fixed-contract diagnostic that holds the selector's setup fixed and changes one decision slot at a time.
  arXiv  Detail & Related papers  (2026-05-07T00:36:59Z)
• ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection [8.362927764080203]
  We propose a training-free scorer that ranks tokens by Euclidean distance to the key centroid.<n>ManifoldKV achieves 95.7% accuracy at 4K-16K contexts with 20% compression.<n> WindowedManifoldKV restores accuracy to 84.3% at 25% compression, a 49-point recovery over global L2 and +3.2 points over KeyDiff.
  arXiv  Detail & Related papers  (2026-02-09T07:28:55Z)
• One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache [38.49582847975703]
  We propose DynaKV, a novel post-training framework for low-rank KV cache compression.<n>Our method consistently outperforms existing state-of-the-art compression techniques.<n>When integrated with SnapKV, DynaKV retains only 6% of the KV cache while maintaining 94% of the baseline performance on the LongBench benchmark.
  arXiv  Detail & Related papers  (2026-02-03T13:20:36Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• SBEST: Spectrum-Based Fault Localization Without Fault-Triggering Tests [17.90798133817018]
  This study investigates the feasibility of using stack traces from crash reports as proxies for fault-triggering tests in Spectrum-Based Fault localization.<n>We propose SBEST, a novel approach that integrates stack trace information with test coverage data to perform fault localization when fault-triggering tests are missing.
  arXiv  Detail & Related papers  (2024-05-01T15:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.