StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving

• URL: http://arxiv.org/abs/2603.28795v1
• Date: Tue, 24 Mar 2026 17:19:26 GMT
• Title: StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving
• Abstract summary: We present StepCache, a backend-agnostic step-level reuse layer that segments outputs into ordered steps.<n>StepCache regenerates only failing regions via selective patching.<n>It reduces mean latency from 2.13 s to 0.67 s, median latency from 2.42 s to 0.01 s, and p95 latency from 3.38 s to 3.30 s.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We address LLM serving workloads where repeated requests share a common solution structure but differ in localized constraints, such as output schema, variable names, or numeric constants. Prior caching approaches typically reuse either full responses (semantic caching) or model-internal KV/prefix states, which are respectively brittle under partial changes or tightly coupled to specific backends. We present StepCache, a backend-agnostic step-level reuse layer that segments outputs into ordered steps, retrieves the best-matching cached request, verifies steps using lightweight task-aware checks, and regenerates only failing regions via selective patching. StepCache additionally supports strict structured-output enforcement for JSON, including single-step extraction, required-key constraints, and one-shot repair, as well as conservative skip-reuse fallbacks for semantic changes. For linear equations, StepCache promotes verification into correction via a bounded repair loop with a deterministic fallback that guarantees correctness when the backend model fails. In a CPU-only perturbation-heavy micro-benchmark on math and JSON variants, averaged over three seeds, StepCache reduces mean latency from 2.13 s to 0.67 s, median latency from 2.42 s to 0.01 s, and p95 latency from 3.38 s to 3.30 s. It also reduces total token usage from 36.1k to 27.3k and improves end-to-end correctness from 72.5% to 100% under task-specific checks and a stitched-output integrity check. Across requests, 79.7% take the reuse-only fast path, 5.4% require patching, and 14.9% trigger skip-reuse.

Related papers

• Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache [6.185334606321749]
  External cache transfers can succeed while a hybrid language model resumes from an inconsistent state.<n>We apply an existing checkpoint-alignment principle to a 45-layer GLM-5.3-Flash model.
  arXiv  Detail & Related papers  (2026-09-14T04:46:09Z)
• Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries [6.185334606321749]
  We study two single-GPU 27B vLLM replicas sharing a 256 GiB LMCache pool.<n>Six-turn synthetic sessions alternating replicas improve by approximately 35% and 45% at initial contexts of 32k and 128k.
  arXiv  Detail & Related papers  (2026-09-14T04:37:44Z)
• KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints [14.86076891340077]
  KVShare-cache reuse across prompt contexts and model checkpoints on retrieved chunks and agent reports.<n>We find that correcting positions, which needs no recomputation, is enough until a question needs several sources at once.<n> KVArena-cache scores every method by the fraction of the gap it recovers between no cache and full recomputation.
  arXiv  Detail & Related papers  (2026-09-09T14:53:45Z)
• Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving [0.0]
  Prefix caching, in which a serving engine reuses the key and value tensors of a shared prompt prefix across requests, is enabled by default in the major open-source stacks.<n>We run an eighty-episode multi-turn agentic tool-use workload with caching enabled and disabled across two engines and four weight formats.<n>We find that repeated execution was bit-identical in every configuration, 0 of 800 episodes, which bounds other sources of nondeterminism at 0.5 percent.
  arXiv  Detail & Related papers  (2026-09-04T05:27:34Z)
• KeyPooling: Measuring Where LLM API Relay Paths Collapse Prompt Cache Isolation [53.952294884822955]
  Large language model (LLM) API relays authenticate customers separately but often forward requests through shared provider credentials.<n>We present KeyPooling, a measurement method that traces customer identity through cache lookup and write.<n>We derive a defense contract: every customer must enter a provider-enforced domain.
  arXiv  Detail & Related papers  (2026-08-18T08:12:34Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs [0.0]
  We present FreshCache, a three-tier semantic cache that treats cache reuse as a risk-constrained temporal inference problem.<n>FreshCache approves reuse only when that probability falls below a per-tier error budget across answers, URL lists, and page content.<n>At the 24-hour evaluation window, FreshCache_MLP achieves 97% search API savings at 0.1% hash-based stale error.
  arXiv  Detail & Related papers  (2026-07-05T12:47:36Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)
• Accelerating Diffusion Transformer via Error-Optimized Cache [17.666577782052205]
  Diffusion Transformer (DiT) is a crucial method for content generation.<n>Existing caching methods accelerate generation by reusing DiT features from the previous time step and skipping calculations in the next.<n>We propose the textbfError-textbfOptimized textbfCache (textbfEOC) to solve this problem.
  arXiv  Detail & Related papers  (2025-01-31T15:58:15Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• MeanCache: User-Centric Semantic Caching for LLM Web Services [8.350378532274405]
  Caching is a natural solution to reduce inference costs on repeated queries.<n>This paper introduces MeanCache, a user-centric semantic cache for LLM-based services.<n>MeanCache identifies semantically similar queries to determine cache hit or miss.
  arXiv  Detail & Related papers  (2024-03-05T06:23:50Z)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference [78.65321721142624]
  We focus on a memory bottleneck imposed by the key-value ( KV) cache. Existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs. We propose LESS, a simple integration of a constant sized cache with eviction-based cache methods.
  arXiv  Detail & Related papers  (2024-02-14T18:54:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.