RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

• URL: http://arxiv.org/abs/2605.06675v1
• Date: Wed, 22 Apr 2026 02:31:58 GMT
• Title: RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
• Abstract summary: A natural idea is to allocate more bits to important heads and fewer to the rest.<n>Applying one quantizer's distortion model to another inverts the allocation order and makes performance worse than uniform quantization.<n>RateQuant fits a per-quantizer distortion model from a small calibration set, then solves the resulting bit-allocation problem in closed form.
• Score: 3.307797786204237
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving. Quantizing the KV cache to fewer bits reduces this cost, yet all current quantizers assign the same bit-width to every attention head, ignoring the large variation in head importance. A natural idea is to allocate more bits to important heads and fewer to the rest. We show, however, that such mixed-precision allocation has a hidden pitfall: each quantizer follows a different distortion curve D(b)=alpha*beta^{-b}, and the decay rate beta varies from 3.6 to 5.3 across quantizer designs. Applying one quantizer's distortion model to another inverts the allocation order and makes performance worse than uniform quantization. We call this failure mode distortion model mismatch and propose RateQuant to resolve it. RateQuant fits a per-quantizer distortion model from a small calibration set, then solves the resulting bit-allocation problem in closed form via reverse waterfilling from rate-distortion theory. On Qwen3-8B at 2.5 average bits, calibrated RateQuant reduces KIVI's perplexity from 49.3 to 14.9 (70% reduction) and improves QuaRot by 6.6 PPL. The entire calibration takes 1.6 s on a single GPU and adds zero overhead at inference time.

Related papers

• SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference [3.2938220347051175]
  We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully.<n>We show that replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) prunes the cliff in every protocol tested.
  arXiv  Detail & Related papers  (2026-08-28T22:19:47Z)
• KV Cache Compression Through the Lens of Transform Coding [9.884224071643606]
  We introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion.<n>Our method achieves near-lossless accuracy at approximately $5.8times$ compression, whereas each baseline degrades in at least some settings.
  arXiv  Detail & Related papers  (2026-08-14T11:08:01Z)
• HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models [0.0]
  HyperQuant is a unified post-training quantization pipeline for the weights and the KV cache of large language and diffusion transformers.<n>It outperforms the recent HIGGS scheme at every operating point from 3 to 5 bits per scalar (bps) on weights, and beats both TurboQuant and OCTOPUS on KV quantization down to 1.7 bps.<n>It compresses the linear weights 3.9x and the KV cache 3.79x at near-lossless quality.
  arXiv  Detail & Related papers  (2026-06-22T14:30:47Z)
• AXELRAM: Quantize Once, Never Dequantize [0.0]
  AXELRAM computes attention scores directly from quantized KV cache indices without dequantization.<n>We trace the root cause to layer-wise norm and propose a gradient-free sign pattern selection (200 candidates, 8 calibration samples, one-time) that eliminates catastrophic spikes with zero additional hardware cost.
  arXiv  Detail & Related papers  (2026-04-03T02:03:38Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• PatternKV: Flattening KV Representation Expands Quantization Headroom [37.83913102876393]
  KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference.<n> KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness.<n>We show that the K cache maintains a stable structure that evolves gradually with context, while the V cache carries latent semantic regularities.
  arXiv  Detail & Related papers  (2025-10-05T12:09:14Z)
• AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models [27.605195979962474]
  Quantization has emerged as an effective and lightweight solution to reduce the memory footprint of the KV cache in Large Language Models.<n>We propose AnTKV, a dual-stage framework that leverages anchor token-aware vector quantization to compress the KV cache.<n>Experiments demonstrate that AnTKV matches or surpasses prior methods at 4-bit, and significantly reduce perplexity under ultra-low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-24T10:45:48Z)
• TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate [13.14434628836727]
  Vector quantization aims to quantize high-dimensional Euclidean vectors while minimizing distortion in their geometric structure.<n>We propose TurboQuant to address both mean-squared error (MSE) and inner product distortion.<n>Our data-oblivious algorithms, suitable for online applications, achieve near-optimal distortion rates.
  arXiv  Detail & Related papers  (2025-04-28T15:05:35Z)
• Quantize What Counts: More for Keys, Less for Values [63.51476878610841]
  Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value ( KV) cache.<n>This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models.
  arXiv  Detail & Related papers  (2025-02-20T22:24:27Z)
• Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression [87.5604418100301]
  Key-value( KV) caching is an important technique to accelerate the inference of large language models. Existing methods often compromise precision or require extra data for calibration. We introduce textbfDecoQuant, a novel data-free low-bit quantization technique based on tensor decomposition methods.
  arXiv  Detail & Related papers  (2024-05-21T08:35:10Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)
• The case for 4-bit precision: k-bit Inference Scaling Laws [75.4335600212427]
  Quantization methods reduce the number of bits required to represent each parameter in a model. The final model size depends on both the number of parameters of the original model and the rate of compression. We run more than 35,000 zero-shot experiments with 16-bit inputs and k-bit parameters to examine which quantization methods improve scaling for 3 to 8-bit precision.
  arXiv  Detail & Related papers  (2022-12-19T18:48:33Z)
• HAWQV3: Dyadic Neural Network Quantization [73.11579145354801]
  Current low-precision quantization algorithms often have the hidden cost of conversion back and forth from floating point to quantized integer values. We present HAWQV3, a novel mixed-precision integer-only quantization framework.
  arXiv  Detail & Related papers  (2020-11-20T23:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.