HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization

• URL: http://arxiv.org/abs/2605.03562v2
• Date: Thu, 07 May 2026 03:08:59 GMT
• Title: HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
• Abstract summary: KV-cache quantizers usually optimize storage-space reconstruction, even though attention reads keys through logits and values through attention-weighted readout.<n>We argue that persistent cache error should be measured in model-visible coordinates.<n>For keys, the visible object is score error modulo constant shifts; this yields HeadQ, a key-side method that stores a low-rank residual side code in a calibration-learned query basis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: KV-cache quantizers usually optimize storage-space reconstruction, even though attention reads keys through logits and values through attention-weighted readout. We argue that persistent cache error should be measured in model-visible coordinates. For keys, the visible object is score error modulo constant shifts; this yields HeadQ, a key-side method that stores a low-rank residual side code in a calibration-learned query basis and applies it as an additive logit correction. For values, fixed-attention readout gives an $A^2$-weighted token-distortion surrogate. Across six models, Fisher/score-space error predicts attention KL far better than raw key MSE; same-budget counterexamples, null-space interventions, query-PCA controls, and wrong-sign HeadQ falsify storage-MSE alternatives. Matched Pythia checkpoints localize the main anomaly to a small-model low-entropy route-flip boundary. In K-only WikiText-103 decode experiments with dense values, HeadQ removes roughly $84$--$94\%$ of the excess perplexity on the strongest 2-bit rows; in an auxiliary full-KV 2-bit composition, HeadQ plus an $A^2$ value policy improves all six models.

Related papers

• Nishimori Threshold Estimation for Bayesian Inference and $\mathbb{Z}_q$ Surface Code Decoding [0.4910937238451485]
  We introduce an analytical estimate of error thresholds falling into the Nishimori universality class via a Fourier--Walsh projection scheme.<n>We find that the threshold values not only accurately agree with recent decohered-$mathbb Z_q$-toric-code numerics, but are found to satisfy the Gilbert-Varshamov self-dual entropy relation.<n>Our approach thereby points to a deeper connection between the clean and Born-disordered models, while allowing for instant analytical estimates of error thresholds for a variety of stabilizer codes.
  arXiv  Detail & Related papers  (2026-07-20T18:00:00Z)
• AVQ-Attention: Adaptive Vector-Quantized Attention [56.79217681896275]
  We propose Adaptive Vector-Quantized (AVQ) Attention, which adaptively allocates codebook capacity based on attention importance.<n>Our approach maintains $mathcalO(MN)$ complexity while achieving improved accuracy-efficiency trade-offs compared to fixed-codebook VQ-attention.
  arXiv  Detail & Related papers  (2026-07-14T14:02:22Z)
• QK-Normed MLA: QK normalization without full key caching [65.51862319225098]
  We show that QK normalization is a practical stabilization option for MLA models without requiring full-key caching.<n>In our 400M runs trained for up to 100B tokens, QK-Normed MLA achieves lower training loss and better downstream accuracy than QK clipping.
  arXiv  Detail & Related papers  (2026-06-15T07:16:30Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks [4.530056625101704]
  We introduce KVarN, a calibration-free KV-cache quantizer that applies a Hadamard rotation followed by a dual-scaling variance normalization.<n>We find that this combination fixes outlying token-scale errors and substantially reduces error accumulation over existing baselines.<n>KVarN establishes a new state-of-theart for KV-cache quantization on generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision.
  arXiv  Detail & Related papers  (2026-06-02T10:34:56Z)
• Linear Predictability of Attention Heads in Large Language Models [0.5833117322405447]
  We show that pretrained Transformers exhibit a pervasive inter-head linear structure.<n>We exploit this by caching only reference-head KV states and reconstructing the remaining heads on the fly.
  arXiv  Detail & Related papers  (2026-03-04T09:04:03Z)
• Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection [15.230462656535034]
  Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = dmodel$)<n>Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary.<n>We argue that selection is an inherently lower-dimensional operation than value transfer, requiring only $BigO(log N) dimensions to distinguish among $N relevant patterns.
  arXiv  Detail & Related papers  (2026-02-16T23:45:39Z)
• CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning [62.56541355300587]
  We introduce a general test-time calibration framework that adaptively modifies the model toward high-reward reasoning paths.<n>Within this framework, we propose CarBoN, a two-phase method that first explores the solution space and then learns a calibration of the logits.<n>Experiments on MATH-500 and AIME-2024 show that CarBoN improves efficiency, with up to $4times$ fewer rollouts to reach the same accuracy.
  arXiv  Detail & Related papers  (2025-10-17T14:04:37Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Causal Attention with Lookahead Keys [52.63961482746826]
  In standard causal attention, each token's query, key, and value (QKV) are static and encode only preceding context.<n>We introduce CAuSal aTtention with Lookahead kEys (CASTLE), an attention mechanism that continually updates each token's keys as the context unfolds.
  arXiv  Detail & Related papers  (2025-09-09T00:15:23Z)
• Quantize What Counts: More for Keys, Less for Values [63.51476878610841]
  Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value ( KV) cache.<n>This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models.
  arXiv  Detail & Related papers  (2025-02-20T22:24:27Z)
• Continuous-Variable Quantum Key Distribution with key rates far above the PLOB bound [0.7918886297003017]
  We provide an analysis of the secret key rate for the case of Gaussian collective attacks. We obtain secret key rates far above the Devetak-Winter value $I(X;Y) - I(E;Y)$, which is the upper bound in the case of one-way error correction.
  arXiv  Detail & Related papers  (2024-02-07T11:40:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.