RoPE-Aware Bit Allocation for KV-Cache Quantization

• URL: http://arxiv.org/abs/2606.24033v1
• Date: Tue, 23 Jun 2026 00:17:48 GMT
• Title: RoPE-Aware Bit Allocation for KV-Cache Quantization
• Abstract summary: Block-GTQ is a bit allocator for key-cache quantization built on TurboQuant-MSE.<n>It better preserves RoPE query-key logits on a ten-model diagnostic panel.<n>It runs 1.34x faster than fp16 FlashAttention2 at 128K context.
• Score: 52.099459337231345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector. Under RoPE, however, a key's contribution to a future attention logit decomposes into a position-dependent sum over two-dimensional frequency blocks. This makes key-cache quantization a block-wise bit-allocation problem: high-energy RoPE blocks are more sensitive to quantization error and should receive more bits. We introduce Block-GTQ, a RoPE-aware bit allocator for key-cache quantization built on TurboQuant-MSE(TQ-MSE). For each layer and KV head, Block-GTQ computes a label-free energy score for each RoPE block and greedily allocates integer bit widths by marginal gain. Under matched K/V bit budgets, Block-GTQ better preserves RoPE query-key logits on a ten-model diagnostic panel, cutting per-layer MAE by 32-80% at 2 and 3 b/dim K-only quantization and winning all 367/367 layer comparisons against uniform TQ-MSE. These fidelity gains translate to stronger downstream long-context retrieval, understanding, and reasoning. At K2V2 on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task NIAH average from 70.6 to 97.4, and the LongBench-EN average from 36.87 to 53.31. On AIME 2024/2025 with DeepSeek-R1-Distill-Qwen-7B, without an fp16 recent-key buffer, Block-GTQ at K3V2 scores 51.7/37.5, close to fp16's 54.2/37.9, whereas uniform TQ-MSE collapses to 0.0/0.0. We further implement a packed-cache serving path. On a single H800 GPU with Qwen2.5-3B-Instruct, packed K3V3 achieves 3.24x KV-cache compression with fp16-comparable quality, runs 1.34x faster than fp16 FlashAttention2 at 128K context, reduces peak memory from 56.31 GB to 19.85 GB, and remains feasible at 256K and 512K where fp16 OOMs. Code is available at https://github.com/JIA-Lab-research/blockgtq.

Related papers

• Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention [0.0]
  Minima-KV is a retention-preserving hierarchy for mixed-format paged attention.<n>Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3.<n> format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax state.
  arXiv  Detail & Related papers  (2026-08-24T21:21:58Z)
• RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention [0.0]
  Large mixture-of-experts (MoE) language models with 26-120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures.<n>We present RotaryQuant, a three-axis compression system that addresses all three.
  arXiv  Detail & Related papers  (2026-08-08T11:58:56Z)
• FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference [0.0]
  FreqDepthKV is an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals.<n>A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits.<n>Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets.
  arXiv  Detail & Related papers  (2026-07-07T17:26:28Z)
• NestedKV: Nested Memory Routing for Long-Context KV Cache Compression [35.62789874560166]
  NestedKV is a key-only KV cache compression method inspired by the Continuum Memory System in Nested Learning.<n>It scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner.
  arXiv  Detail & Related papers  (2026-05-26T08:14:39Z)
• XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference [0.0]
  XFP determines codebook size, outlier budget, and packing per layer automatically.<n>XFP reaches 138 tok/s single-stream decode on workstation hardware.<n>For models that do not fit in the target memory envelope, we present the H-Process.
  arXiv  Detail & Related papers  (2026-05-14T13:52:31Z)
• LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformers [0.0]
  Current quantization methods compress storage but fail to reduce bandwidth as attention calculation requires dequantizing keys from INT4/INT8 to FP16 before use.<n>We propose LOOKAT, which applies product quantization and asymmetric distance computation to transformer architecture by decomposing key vectors into subspaces.
  arXiv  Detail & Related papers  (2026-01-15T07:54:07Z)
• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference [48.40143137402824]
  Multi-Head Latent Attention (MLA) compresses key-value states into a low-rank latent vector, caching only this vector to reduce memory.<n>In tensor parallelism (TP), however, attention heads are computed across multiple devices, and each device must load the full cache.<n>We propose-Parallel Latent Attention (TPLA), a scheme that partitions both the latent representation and each head's input dimension across devices, performs attention independently per shard, and then combines results with an all-reduce.
  arXiv  Detail & Related papers  (2025-08-21T15:25:40Z)
• CommVQ: Commutative Vector Quantization for KV Cache Compression [50.37946553931796]
  We propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference.<n>We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache.<n>Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook.
  arXiv  Detail & Related papers  (2025-06-23T17:50:11Z)
• KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head [0.8158530638728501]
  We introduce KV-Compress, a novel compression method that evicts contiguous KV blocks within a PagedAttention framework. Our method achieves state-of-the-art performance on LongBench for both Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct while lowering the total number of compressed KVs by 4x. Evaluations on Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct-FP8 achieve compression rates up to 8x with negligible impact on performance, and up to 64x while retaining over 90% of full-cache performance
  arXiv  Detail & Related papers  (2024-09-30T19:09:13Z)
• PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference [57.53291046180288]
  Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference. We propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache.
  arXiv  Detail & Related papers  (2024-05-21T06:46:37Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.