Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries

• URL: http://arxiv.org/abs/2609.15021v1
• Date: Mon, 14 Sep 2026 04:37:44 GMT
• Title: Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries
• Abstract summary: We study two single-GPU 27B vLLM replicas sharing a 256 GiB LMCache pool.<n>Six-turn synthetic sessions alternating replicas improve by approximately 35% and 45% at initial contexts of 32k and 128k.
• Score: 6.185334606321749
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Shared host-memory caching can avoid repeated prefill when a request moves between inference replicas. Its usefulness depends on both correct state transfer and lost prefix locality. We study two single-GPU 27B vLLM replicas sharing a 256 GiB LMCache pool. After adopting an existing packed-page patch, we isolate a raw-pointer fallback that omits the dependency on the current CUDA stream. Controlled byte tests fail under an imposed delay and pass when the dependency is restored; the existing mixed allocator provides a working deployment path. Full-pool allocation checks and service regression complete the validation. A four-block OFF-ON-ON-OFF comparison contains 768 measured requests within two block pairs. Median cross-replica time to first content token falls from 31.715 to 0.605 seconds at 128k input and from 92.047 to 0.790 seconds at 256k. Six-turn synthetic sessions alternating replicas improve by approximately 35% and 45% at initial contexts of 32k and 128k, while fixed placement shows little benefit. This engineering case study identifies practical validation steps and the locality conditions in which shared caching pays off.

Related papers

• Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache [6.185334606321749]
  External cache transfers can succeed while a hybrid language model resumes from an inconsistent state.<n>We apply an existing checkpoint-alignment principle to a 45-layer GLM-5.3-Flash model.
  arXiv  Detail & Related papers  (2026-09-14T04:46:09Z)
• Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving [0.0]
  Prefix caching, in which a serving engine reuses the key and value tensors of a shared prompt prefix across requests, is enabled by default in the major open-source stacks.<n>We run an eighty-episode multi-turn agentic tool-use workload with caching enabled and disabled across two engines and four weight formats.<n>We find that repeated execution was bit-identical in every configuration, 0 of 800 episodes, which bounds other sources of nondeterminism at 0.5 percent.
  arXiv  Detail & Related papers  (2026-09-04T05:27:34Z)
• Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference [0.0]
  Key-value (KV) cache is the primary throughput optimization in modern large language model (LLM) inference.<n>Three published attacks exploit it, reaching up to 100% attack success rate against unprotected vLLM and SG.<n>We present KVGov, a governance layer addressing all three attack families' prefix-cache paths.
  arXiv  Detail & Related papers  (2026-08-10T07:47:57Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving [0.0]
  We present StepCache, a backend-agnostic step-level reuse layer that segments outputs into ordered steps.<n>StepCache regenerates only failing regions via selective patching.<n>It reduces mean latency from 2.13 s to 0.67 s, median latency from 2.42 s to 0.01 s, and p95 latency from 3.38 s to 3.30 s.
  arXiv  Detail & Related papers  (2026-03-24T17:19:26Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)
• Accelerating Deep Learning Classification with Error-controlled Approximate-key Caching [72.50506500576746]
  We propose a novel caching paradigm, that we named approximate-key caching. While approximate cache hits alleviate DL inference workload and increase the system throughput, they however introduce an approximation error. We analytically model our caching system performance for classic LRU and ideal caches, we perform a trace-driven evaluation of the expected performance, and we compare the benefits of our proposed approach with the state-of-the-art similarity caching.
  arXiv  Detail & Related papers  (2021-12-13T13:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.