Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

• URL: http://arxiv.org/abs/2609.11744v1
• Date: Thu, 10 Sep 2026 16:00:35 GMT
• Title: Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
• Abstract summary: py-kvcache is a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading.<n>At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x.<n>With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation.
• Score: 0.22940141855172033
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.

Related papers

• InferScale: GPU-Native KV Injection for Personalized LLM Serving [2.1906014825510307]
  We present InferScale, a GPU-native memory system that replaces repeated prompt prefilling with reusable KV state.<n>InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, and retrieves relevant facts at serving time.<n> Context-Window encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV.
  arXiv  Detail & Related papers  (2026-07-29T16:18:22Z)
• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs [12.353502602473695]
  Key-Value (KV) cache plays a crucial role in accelerating inference in large language models.<n>We propose a novel KV cache management strategy, which integrates semantic token clustering with PagedAttention.<n>With a 256-token budget, IceCache maintains 99% of the original accuracy achieved by the full KV cache model.
  arXiv  Detail & Related papers  (2026-04-12T09:02:20Z)
• Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts [68.79341332280062]
  Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time.<n>We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier.<n>Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint.
  arXiv  Detail & Related papers  (2026-02-02T13:52:40Z)
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference [27.24239725255976]
  We present LMCache, the first and so far the most efficient open-source KV caching solution.<n>It extracts and stores KV caches generated by modern LLM engines and shares the KV caches across engines and queries.<n>We show that combining LMCache with vLLM achieves up to 15x improvement in throughput across diverse workloads.
  arXiv  Detail & Related papers  (2025-10-08T00:15:04Z)
• dKV-Cache: The Cache for Diffusion Language Models [53.85291644298835]
  Diffusion Language Models (DLMs) have been seen as a promising competitor for autoregressive language models.<n>We propose a KV-cache-like mechanism, delayed KV-Cache, for the denoising process of DLMs.<n>Our approach is motivated by the observation that different tokens have distinct representation dynamics throughout the diffusion process.
  arXiv  Detail & Related papers  (2025-05-21T17:32:10Z)
• SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs [44.41154292836592]
  We propose SpeCache, which offloads the complete KV cache and dynamically fetches KV pairs back in each decoding step.<n> Experiments on LongBench and Needle-in-a-Haystack benchmarks verify that SpeCache effectively reduces VRAM usage.
  arXiv  Detail & Related papers  (2025-03-20T14:01:56Z)
• Compute Or Load KV Cache? Why Not Both? [6.982874528357836]
  Cake is a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel.<n> Cake achieves on average 2.6x reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods.
  arXiv  Detail & Related papers  (2024-10-04T01:11:09Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference [78.65321721142624]
  We focus on a memory bottleneck imposed by the key-value ( KV) cache. Existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs. We propose LESS, a simple integration of a constant sized cache with eviction-based cache methods.
  arXiv  Detail & Related papers  (2024-02-14T18:54:56Z)
• CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving [31.766738294505767]
  CacheGen is a fast context-loading module for large language models. Uses a custom tensor encoder to encode a KV cache into compact bitstream representations. adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth.
  arXiv  Detail & Related papers  (2023-10-11T07:08:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.