Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap

• URL: http://arxiv.org/abs/2608.23658v1
• Date: Mon, 24 Aug 2026 15:49:39 GMT
• Title: Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap
• Abstract summary: elastic KV cache lends the reserve to the KV pool during decode and returns it before prefill.<n>A static commit of same memory is unsafe crashing on prefill bursts, which makes the dynamic toggle necessary.<n>It only pays off if a small prefill chunk size badly hurts prefill latency.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An LLM serving engine sizes its key-value (KV) cache once, at startup, permanently setting aside a reserve for the worst-case prefill activation. During decode-dominant phases that reserve sits idle, yet it cannot be handed to the KV pool because it is exactly the memory a large prefill needs. We ask whether this reserve is reclaimable, and build a mechanism to test it. Our elastic KV cache lends the reserve to the KV pool during decode and returns it before prefill, driven by the scheduler's one-step-ahead view of the next batch. It is pure userspace on the CUDA virtual-memory path: two physical handles mapped into one contiguous virtual range per layer, so the attention kernel is unchanged and no driver patch is required. It decommits in a few milliseconds and recommits in tens of milliseconds, works with CUDA graphs and prefix caching, and never triggers an out-of-memory event. A static commit of the same memory is unsafe, crashing on prefill bursts, which makes the dynamic toggle necessary. Having built the mechanism, we test the premise it rests on and report an honest negative result. It only pays off if a small prefill chunk size badly hurts prefill latency. In a controlled experiment injecting long prompts into a live decode load, that penalty is small (median time-to-first-token differs by about 1% between chunk sizes of 8192 and 32768 tokens), because prefill is compute bound and decode consumes only about one token per sequence per step. Simply lowering max_num_batched_tokens recovers more KV than the controller does, at nearly equal latency. The reserve also dilutes under tensor parallelism, from 16% of KV at TP1 to 2.7% at TP4. We state precisely when reclaiming the reserve could still help, and release the mechanism as a reusable userspace elastic-VMM allocator.

Related papers

• Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs [0.22940141855172033]
  py-kvcache is a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading.<n>At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x.<n>With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation.
  arXiv  Detail & Related papers  (2026-09-10T16:00:35Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• InferScale: GPU-Native KV Injection for Personalized LLM Serving [2.1906014825510307]
  We present InferScale, a GPU-native memory system that replaces repeated prompt prefilling with reusable KV state.<n>InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, and retrieves relevant facts at serving time.<n> Context-Window encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV.
  arXiv  Detail & Related papers  (2026-07-29T16:18:22Z)
• PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs [0.8460698440162889]
  We present PersistentKV, a block-table decode attention engine and page-aware scheduling study.<n>Per PersistentKV maps work by KV-head group, executes directly over native page tables, and adds a compact workqueue schedule.<n>We show that five held-out seeds improve mean wall decode-token throughput by 1.04x to 1.08x on bimodal, uniform, and Zipf-like workloads.
  arXiv  Detail & Related papers  (2026-06-25T06:56:43Z)
• Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving [0.5076419064097734]
  We study the opposite regime: low-latency, small-batch, on-device physical-AI serving.<n>We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state.<n>This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries.
  arXiv  Detail & Related papers  (2026-06-18T17:49:36Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• MEPIC: Memory Efficient Position Independent Caching for LLM Serving [16.99046229452175]
  We present a memory-efficient system that enables chunk KV reuse across positions, requests, and batches.<n>MePIC aligns chunk KV to paged storage, shifts recomputation from token- to block-level so only the first block is request-specific.
  arXiv  Detail & Related papers  (2025-12-18T18:04:01Z)
• CommVQ: Commutative Vector Quantization for KV Cache Compression [50.37946553931796]
  We propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference.<n>We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache.<n>Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook.
  arXiv  Detail & Related papers  (2025-06-23T17:50:11Z)
• Compute Or Load KV Cache? Why Not Both? [6.982874528357836]
  Cake is a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel.<n> Cake achieves on average 2.6x reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods.
  arXiv  Detail & Related papers  (2024-10-04T01:11:09Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference [78.65321721142624]
  We focus on a memory bottleneck imposed by the key-value ( KV) cache. Existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs. We propose LESS, a simple integration of a constant sized cache with eviction-based cache methods.
  arXiv  Detail & Related papers  (2024-02-14T18:54:56Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.