Learning Agent Execution for KV-Cache Management in Agentic Serving

• URL: http://arxiv.org/abs/2608.14624v1
• Date: Thu, 16 Jul 2026 22:28:19 GMT
• Title: Learning Agent Execution for KV-Cache Management in Agentic Serving
• Abstract summary: We present CacheScout, an agent-aware KV-cache runtime layer for multi-agent LLM serving.<n>The key insight is that future KV-cache reuse is governed by agent execution semantics rather than cache recency alone.<n>We show that CacheScout improves KV-cache hit rate by 10-18 percentage points, reduces mean TTFT by 18-45%, lowers mean per-turn latency by 29-38%, and increases peak throughput by up to 57%.
• Score: 25.021363822677515
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents. Across these workflows, every agent repeatedly executes a fixed context consisting of system prompts, tool definitions, and few-shot examples, creating substantial opportunities for KV-cache reuse. Existing LLM serving systems, however, manage KV-cache reactively using prefix caching and recency-based replacement, causing reusable agent contexts to be evicted before their next invocation and forcing repeated recomputation. We present CacheScout, an agent-aware KV-cache runtime layer for multi-agent LLM serving. The key insight is that future KV-cache reuse is governed by agent execution semantics rather than cache recency alone. CacheScout captures these semantics by learning agent execution transitions online, without requiring predefined workflow graphs or offline training, and uses the learned execution model to guide both cache eviction and proactive prefetching while leaving the serving critical path unchanged. We implement CacheScout on top of vLLM. Across representative real-world multi-agent workloads, CacheScout improves KV-cache hit rate by 10-18 percentage points, reduces mean TTFT by 18-45%, lowers mean per-turn latency by 29-38%, and increases peak throughput by up to 57%. These benefits also generalize to larger models, reducing TTFT by up to 54% while sustaining 37% higher throughput.

Related papers

• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Recency/Frequency Adaptive KV Caching for Large Language Model Serving [7.261125311665803]
  Key-value ( KV) caching is a powerful technique for accelerating large language model inference and generation.<n>Inference workloads are large and diverse, which makes them difficult to cache effectively.<n>We integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks.
  arXiv  Detail & Related papers  (2026-06-19T09:05:01Z)
• A Policy-Driven Runtime Layer for Agentic LLM Serving [6.975319765760388]
  Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them.<n>We argue this seam is best addressed by an architectural change rather than point fixes.<n>Preliminary results on five real multi-agent workloads show +13 to +37 pp cache hit-rate lift, 12% to 29% lower mean TTFT, and 6% to 14% higher throughput over an unmodified serving stack.
  arXiv  Detail & Related papers  (2026-05-26T22:38:34Z)
• PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design [5.476189551376108]
  We present PRISM, which co-designs a query-aware scheduler (QAS) with a demand-aware rad tree (DART) to align request admission with exact- KV retention.<n>Our evaluation results show that, versus the strongest baseline, PRISM reduces average per-QPS P99 TTFT by 23.3% and 37.1% while increasing exact- KV-cache hit rate by 5.9 and 12.2 percentage points on 4B and 13B models, respectively.
  arXiv  Detail & Related papers  (2026-05-09T00:48:25Z)
• AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agents [3.275342475988451]
  Embodied AI agents increasingly rely on large language models (LLMs) for planning, yet per-step LLM calls impose severe latency and cost.<n>We introduce AgenticCache, a framework that reuses cached plans to avoid per-step LLM calls.<n>AgenticCache improves task success rate by 22% on average across 12 configurations, reduces simulation latency by 65%, and lowers token usage by 50%.
  arXiv  Detail & Related papers  (2026-04-27T04:51:15Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems [25.770173970846884]
  KVCOMM is a training-free framework that enables efficient prefilling in multi-agent inference.<n> KVCOMM estimates and adjusts KV-caches for shared content by referencing a pool of cached examples-termed anchors.<n> KVCOMM achieves over 70% reuse rate across diverse multi-agent workloads.
  arXiv  Detail & Related papers  (2025-10-14T18:00:01Z)
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness [75.00019285120878]
  Key-value (KV) caching can mitigate this, but storing the full cache is prohibitive for image-heavy contexts.<n>Existing cache-compression methods are sub-optimal as they do not account for the spatial and temporal redundancy of GUIs.<n>We introduce GUI-KV, a plug-and-play KV cache compression method for GUI agents that requires no retraining.
  arXiv  Detail & Related papers  (2025-10-01T05:37:54Z)
• dKV-Cache: The Cache for Diffusion Language Models [53.85291644298835]
  Diffusion Language Models (DLMs) have been seen as a promising competitor for autoregressive language models.<n>We propose a KV-cache-like mechanism, delayed KV-Cache, for the denoising process of DLMs.<n>Our approach is motivated by the observation that different tokens have distinct representation dynamics throughout the diffusion process.
  arXiv  Detail & Related papers  (2025-05-21T17:32:10Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.