KV Cache Compression Through the Lens of Transform Coding

• URL: http://arxiv.org/abs/2608.14191v1
• Date: Fri, 14 Aug 2026 11:08:01 GMT
• Title: KV Cache Compression Through the Lens of Transform Coding
• Abstract summary: We introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion.<n>Our method achieves near-lossless accuracy at approximately $5.8times$ compression, whereas each baseline degrades in at least some settings.
• Score: 9.884224071643606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attention mechanisms. We prove that, under a white-noise quantization model, the expected attention-aware distortion decomposes into additive key and value contributions that factor across tokens and channels. Building on transform coding and reverse water-filling, which are classical tools from signal processing and rate-distortion theory, we introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion. On Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, evaluated across LongBench, RULER, GSM8K, MMLU-Pro, and MATH-500, our method achieves near-lossless accuracy at approximately $5.8\times$ compression, whereas each baseline degrades in at least some settings.

Related papers

• Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches [25.55174329283001]
  Long-context Transformer inference increasingly relies on KV-cache compression or quantization.<n>We introduce Codec-Gauge, a post-training cache-coordinate layer that learns small channel transforms around existing compression and quantization backends.
  arXiv  Detail & Related papers  (2026-07-10T17:18:25Z)
• DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression [0.0]
  DepthWeave-KV is a token-adaptive cache compression method that factorizes key and value states across neighboring transformer layers.<n> DepthWeave-KV achieves near-full-output task quality with substantially lower memory use.
  arXiv  Detail & Related papers  (2026-07-07T17:29:01Z)
• RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory [3.307797786204237]
  A natural idea is to allocate more bits to important heads and fewer to the rest.<n>Applying one quantizer's distortion model to another inverts the allocation order and makes performance worse than uniform quantization.<n>RateQuant fits a per-quantizer distortion model from a small calibration set, then solves the resulting bit-allocation problem in closed form.
  arXiv  Detail & Related papers  (2026-04-22T02:31:58Z)
• KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction [8.486713415198968]
  We propose KVLinC, a framework to mitigate attention errors introduced by KV cache quantization.<n> KVLinC combines a Hadamard rotation, which reduces quantization error in values, with lightweight linear correction adapters.<n>We show that KVLinC consistently matches or surpasses strong baselines while achieving higher KV-cache compression.
  arXiv  Detail & Related papers  (2025-10-06T21:08:11Z)
• KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding [72.12756830560217]
  Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI.<n>Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value cache during inference has emerged as a primary efficiency bottleneck.<n>By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed.
  arXiv  Detail & Related papers  (2025-07-15T12:52:12Z)
• AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models [27.605195979962474]
  Quantization has emerged as an effective and lightweight solution to reduce the memory footprint of the KV cache in Large Language Models.<n>We propose AnTKV, a dual-stage framework that leverages anchor token-aware vector quantization to compress the KV cache.<n>Experiments demonstrate that AnTKV matches or surpasses prior methods at 4-bit, and significantly reduce perplexity under ultra-low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-24T10:45:48Z)
• Quantize What Counts: More for Keys, Less for Values [63.51476878610841]
  Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value ( KV) cache.<n>This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models.
  arXiv  Detail & Related papers  (2025-02-20T22:24:27Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression [87.5604418100301]
  Key-value( KV) caching is an important technique to accelerate the inference of large language models. Existing methods often compromise precision or require extra data for calibration. We introduce textbfDecoQuant, a novel data-free low-bit quantization technique based on tensor decomposition methods.
  arXiv  Detail & Related papers  (2024-05-21T08:35:10Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.