GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

• URL: http://arxiv.org/abs/2609.03494v1
• Date: Thu, 03 Sep 2026 07:53:05 GMT
• Title: GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
• Abstract summary: Long-output reasoning has made the key--value ( KV) cache a critical memory bottleneck for efficient serving.<n>Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding.<n>We introduce GrowPage, an on-demand KV budgeting framework that treats KV capacity as a runtime resource.
• Score: 17.04616426989979
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding. However, reasoning workloads exhibit substantial demand variation: different requests require different KV capacities, and the attention demand of an individual request evolves during generation. We introduce \textbf{GrowPage}, an on-demand KV budgeting framework that treats KV capacity as a runtime resource. GrowPage maintains lightweight dual-timescale query summaries to capture recent and long-term attention behaviors, and uses their relative attention working sets to estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page when broader demand emerges. By integrating with PagedAttention's page-level memory abstraction, GrowPage preserves continuous batching and prefix caching. Experiments on reasoning benchmarks across multiple models show that GrowPage achieves a superior performance--throughput trade-off over existing approaches.

Related papers

• HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models [10.767250624894665]
  Long-context inference retains a growing key--value (KV) cache during decoding.<n>This bottleneck remains in hybrid language models because their residual global-attention layers can dominate context-dependent cache demand.<n>We introduce HeadWiseKV, a training-free framework that compresses the residual global KV caches of hybrid language models.
  arXiv  Detail & Related papers  (2026-09-02T02:59:11Z)
• PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression [6.397490960170785]
  PuzzleKV is a low-rank compression method for large language models.<n>It decomposes pages within each layer and KV head, computes attention directly over dense and factorized pages, and incrementally compresses newly eligible pages.<n>It achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings.
  arXiv  Detail & Related papers  (2026-08-24T21:30:11Z)
• InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories [6.141047838371132]
  InduceKV is a retrieval-based method that stores each selected training prefix as an attention-ready memory entry.<n>We show that InduceKV consistently improves over PEFT, MoE, replay, and prompt-retrieval baselines under matched memory budgets.
  arXiv  Detail & Related papers  (2026-07-02T10:45:28Z)
• Kwai Summary Attention Technical Report [69.40814939510126]
  Long-context ability, has become one of the most important direction of next-generation Large Language Models.<n>Standard softmax attention exhibits quadratic time complexity with respect to sequence length.<n>We propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts.
  arXiv  Detail & Related papers  (2026-04-27T12:59:53Z)
• LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction [46.60403085398673]
  We introduce LKV (Learned KV Eviction), which formulates KV compression as an end-to-end differentiable optimization problem.<n>LKV achieves state-of-the-art performance on both LongBench and RULER benchmarks at high compression rates.
  arXiv  Detail & Related papers  (2026-04-22T06:35:23Z)
• Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention [58.012620801746046]
  KV cache during the decoding phase has become a critical factor limiting high-concurrency service.<n>This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention.<n>Based on this, we have developed a high-concurrency LLM inference engine, Zipage.
  arXiv  Detail & Related papers  (2026-03-01T14:01:36Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• QAQ: Quality Adaptive Quantization for LLM KV Cache [3.163526369095745]
  A bottleneck in model deployment emerges due to the linear expansion of the Key-Value cache with the context length. We propose QAQ, a Quality Adaptive Quantization scheme for the KV cache.
  arXiv  Detail & Related papers  (2024-03-07T16:42:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.