Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs

• URL: http://arxiv.org/abs/2506.05410v1
• Date: Wed, 04 Jun 2025 16:10:44 GMT
• Title: Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
• Abstract summary: We show a fundamental yet previously overlooked asymmetry in KV caches.<n>While adjacent keys receive similar attention weights (local homogeneity), adjacent values demonstrate distinct heterogeneous distributions.<n>This key-value asymmetry reveals a critical limitation in existing compression methods that treat keys and values uniformly.
• Score: 27.710036447385697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Large Language Models (LLMs) have highlighted the critical importance of extending context length, yet the quadratic complexity of attention mechanisms poses significant challenges for efficient long-context modeling. KV cache compression has emerged as a key approach to address this challenge. Through extensive empirical analysis, we reveal a fundamental yet previously overlooked asymmetry in KV caches: while adjacent keys receive similar attention weights (local homogeneity), adjacent values demonstrate distinct heterogeneous distributions. This key-value asymmetry reveals a critical limitation in existing compression methods that treat keys and values uniformly. To address the limitation, we propose a training-free compression framework (AsymKV) that combines homogeneity-based key merging with a mathematically proven lossless value compression. Extensive experiments demonstrate that AsymKV consistently outperforms existing long-context methods across various tasks and base models. For example, on LLaMA3.1-8B, AsymKV achieves an average score of 43.95 on LongBench, surpassing SOTA methods like H$_2$O (38.89) by a large margin.

Related papers

• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation [0.0]
  We propose an improved low-rank KV cache compression framework.<n>We show that our method reduces the number of Key cache parameters while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2026-07-27T12:08:56Z)
• DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing [30.82607063015274]
  This paper introduces DASH-KV, an innovative acceleration framework that reformulates attention as approximate nearest-neighbor search via asymmetric deep hashing.<n>Experiments on LongBench demonstrate that DASH-KV significantly outperforms state-of-the-art baseline methods while matching the performance of full attention, all while reducing inference complexity from O(N2) to linear O(N)
  arXiv  Detail & Related papers  (2026-04-21T11:33:24Z)
• Latent-Condensed Transformer for Efficient Long Context Modeling [60.72493959155964]
  We propose Latent-Condensed Attention (LCA) for large language models.<n>LCA condenses context within MLA's latent space, where representation is disentangled into semantic latent vectors and positional keys.<n>LCA achieves up to 2.5$times$ prefilling speedup and 90% KV cache reduction at 128K context.
  arXiv  Detail & Related papers  (2026-04-14T08:40:31Z)
• KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging [24.918959010993888]
  Existing methods rely on empirical observations of KV asymmetry and gradient-based Hessian approximations.<n>We introduce KVSlimmer, an efficient algorithm that captures exact Hessian information through a mathematically exact formulation.<n>Experiments across various models and benchmarks demonstrate that KVSlimmer consistently outperforms SOTA methods.
  arXiv  Detail & Related papers  (2026-03-01T04:07:36Z)
• Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction [50.99402504483692]
  We propose a novel gating-based KV cache eviction method for frozen-weight language models.<n>Our approach integrates seamlessly into both the prefill and decoding stages.<n>Experiments show that our method maintains near-lossless performance while evicting up to 70% of the KV cache.
  arXiv  Detail & Related papers  (2026-01-25T03:07:54Z)
• Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models [14.603288559638614]
  textttMixKV is a novel method that mixes importance with diversity for optimized KV cache compression in vision-language models.<n>Under extreme compression, textttMixKV improves baseline methods by an average of textbf5.1% across five multi-modal understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-23T16:17:47Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing [54.34080239841088]
  CommonKV is a training-free method for cross-layer KV cache compression through adjacent parameters sharing.<n>We show that the proposed method consistently outperforms existing low-rank and cross-layer approaches at various compression ratios.
  arXiv  Detail & Related papers  (2025-08-22T06:55:45Z)
• FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression [18.12657364501536]
  FAEDKV is a novel, training-free KV cache compression framework.<n>It preserves both early and recent contextual information.<n>Experiments on LongBench benchmark demonstrate FAEDKV's superiority over existing methods by up to 22%.
  arXiv  Detail & Related papers  (2025-07-26T18:20:25Z)
• TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering [10.427881558469442]
  We introduce TaDA, a training-free recipe for KV cache compression with quantization precision.<n>Our approach yields substantial accuracy improvements for multiple models supporting various context lengths.<n>Our method paves the way for scalable and high-performance reasoning in language models.
  arXiv  Detail & Related papers  (2025-06-05T05:23:38Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [81.81027217759433]
  Large language models (LLMs) are often constrained by the excessive memory required to store the Key-Value ( KV) cache.<n>Recent methods have explored reducing the hidden dimensions of the KV cache, but many introduce additional computation through projection layers.<n>We propose ReCalKV, a post-training KV cache compression method that reduces the hidden dimensions of the KV cache.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important [13.45388421871017]
  LagKV is a KV compression strategy only relying on straight forward comparison among KV themselves.<n>It is a totally attention free method which offers easy integration to the main stream inference platform.<n>Our method outperforms the attention weight based method $HO$ over $50%$ with same compression ratios.
  arXiv  Detail & Related papers  (2025-04-07T03:22:15Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• Optimizing Singular Spectrum for Large Language Model Compression [95.7621116637755]
  We introduce SoCo, a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner.<n>Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores.<n> Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
  arXiv  Detail & Related papers  (2025-02-20T23:18:39Z)
• Streaming Attention Approximation via Discrepancy Theory [11.235024582188288]
  We study the streaming complexity of attention approximation, a key computational primitive underlying token generation.<n>Our main contribution is BalanceKV, a streaming algorithm for $epsilon$-approximating attention computations.
  arXiv  Detail & Related papers  (2025-02-11T17:18:17Z)
• Can LLMs Maintain Fundamental Abilities under KV Cache Compression? [29.510433427184385]
  We present a benchmark KVFundaBench to evaluate the effects of KV cache compression across diverse fundamental language models.<n>We propose ShotKV, a novel compression approach that handles prefill and decoding phases while maintaining shot-level semantic coherence.
  arXiv  Detail & Related papers  (2025-02-04T02:23:06Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods [61.025422435235456]
  We introduce SCBench, a benchmark for evaluating long-context methods from a KV cachecentric perspective.<n>We provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs and Mamba-Attention hybrids.<n>Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n2) pre-filling perform robustly.
  arXiv  Detail & Related papers  (2024-12-13T17:59:52Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization [34.824534775022144]
  We propose Coupled Quantization (CQ) as a technique for KV cache compression. CQ couples multiple key/value channels together to exploit their inter-dependency and encode the activations in a more information-efficient manner. We demonstrate that CQ can preserve model quality with KV cache quantized down to 1-bit.
  arXiv  Detail & Related papers  (2024-05-07T00:25:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.