Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

• URL: http://arxiv.org/abs/2603.04427v1
• Date: Mon, 16 Feb 2026 23:45:39 GMT
• Title: Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
• Abstract summary: Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = dmodel$)<n>Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary.<n>We argue that selection is an inherently lower-dimensional operation than value transfer, requiring only $BigO(log N) dimensions to distinguish among $N relevant patterns.
• Score: 15.230462656535034
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = \dmodel$). Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary. Queries and keys produce scalar attention weights (\emph{selection}), while values carry rich semantic representations (\emph{value transfer}). We argue that selection is an inherently lower-dimensional operation than value transfer, requiring only $\BigO(\log N)$ dimensions to distinguish among $N$ relevant patterns. We validate this hypothesis across seven experiments: (1)~positional selection tasks requiring just 1~dimension per head, (2)~content-based retrieval requiring $\sim\!\log_2 N$ dimensions, (3--4)~WikiText-2 and WikiText-103 language modeling where $\dselect = \dmodel/4$ incurs only 4.3\% perplexity increase while reducing QK parameters by 75\%, (5)~post-training SVD compression of GPT-2, revealing keys to be far more compressible than queries, with lightweight QK fine-tuning recovering nearly all quality loss, (6)~a 125M-parameter LLaMA model confirming identical degradation ratios across architectures, and (7)~Mistral-7B (7.2B parameters), where SVD compression followed by QK fine-tuning achieves 75\% key cache savings at just 2.0\% residual quality cost. For existing models, SVD compression followed by QK fine-tuning (3 epochs on a small fraction of pretraining data) achieves 75\% key cache savings at $<$2\% residual quality cost. For a 7B-parameter model serving 128K context, asymmetric attention saves 25\,GB of KV cache per user, enabling approximately 60\% more concurrent users on the same GPU.

Related papers

• A Universal Context-Reuse Layer for Cross-Model KV Sharing [7.914652203965336]
  We study emphcross-model KV sharing, which translates the KV state produced by a source model into a representation that can be consumed by a different target model.<n>For Qwen2.5-7B $rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59% to 34.48%, a gain of 6.89 percentage points over the native 1.5B baseline.<n>For the cross-family Qwen2.5-1.5B $rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by
  arXiv  Detail & Related papers  (2026-08-31T15:28:17Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Minimal-Intervention KV Retention via Set-Conditioned Diversity [7.208745673318648]
  KV-cache compression at small budgets is a crowded design space spanning cache representation, head-wise routing, compression cadence, decoding behavior, and within-budget scoring.<n>We propose a one-function modification to the TriAttentioncitemao2026triattention retention scorer that replaces argmax-top-$k$ with greedy facility-location-inspired selection.<n>A pre-registered protocol tunes $$ on a frozen development split and confirms on a disjoint held-out split; with $= 0.5$, $$ clears Bonferroni on two of the four (
  arXiv  Detail & Related papers  (2026-05-14T02:50:20Z)
• HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization [0.0]
  KV-cache quantizers usually optimize storage-space reconstruction, even though attention reads keys through logits and values through attention-weighted readout.<n>We argue that persistent cache error should be measured in model-visible coordinates.<n>For keys, the visible object is score error modulo constant shifts; this yields HeadQ, a key-side method that stores a low-rank residual side code in a calibration-learned query basis.
  arXiv  Detail & Related papers  (2026-05-05T09:34:51Z)
• LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformers [0.0]
  Current quantization methods compress storage but fail to reduce bandwidth as attention calculation requires dequantizing keys from INT4/INT8 to FP16 before use.<n>We propose LOOKAT, which applies product quantization and asymmetric distance computation to transformer architecture by decomposing key vectors into subspaces.
  arXiv  Detail & Related papers  (2026-01-15T07:54:07Z)
• Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads [47.05385031325841]
  SkipV1Former is a Transformer variant that uses skip connections from the first layer's Value heads to strengthen representation and reduce KV cache.<n>We show that SkipV1Former delivers consistent reductions of approximately 25 % in KV cache.<n>When combined with YOCO, it cuts KV cache size by nearly 50 % while still improving performance.
  arXiv  Detail & Related papers  (2025-10-19T12:17:42Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• Quantize What Counts: More for Keys, Less for Values [63.51476878610841]
  Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value ( KV) cache.<n>This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models.
  arXiv  Detail & Related papers  (2025-02-20T22:24:27Z)
• QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache [67.84112700032007]
  Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings.<n>In these scenarios, the Key-Value ( KV) cache is the primary bottleneck in terms of both GPU memory and latency.<n>We propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration.
  arXiv  Detail & Related papers  (2025-02-05T20:43:48Z)
• Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models [28.16603647353951]
  AQUA-KV is an adaptive quantization for Key-Value caches that relies on compact adapters.<n>We achieve near-lossless inference at 2-2.5 bits per value with under $1%$ relative error in perplexity and LongBench scores.
  arXiv  Detail & Related papers  (2025-01-31T18:47:42Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.