A Policy-Driven Runtime Layer for Agentic LLM Serving

• URL: http://arxiv.org/abs/2605.27744v1
• Date: Tue, 26 May 2026 22:38:34 GMT
• Title: A Policy-Driven Runtime Layer for Agentic LLM Serving
• Abstract summary: Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them.<n>We argue this seam is best addressed by an architectural change rather than point fixes.<n>Preliminary results on five real multi-agent workloads show +13 to +37 pp cache hit-rate lift, 12% to 29% lower mean TTFT, and 6% to 14% higher throughput over an unmodified serving stack.
• Score: 6.975319765760388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, role, schemas, and dispatch structure but never sees an engine-level event; the serving engine below sees every event but knows nothing about agents. A surprising number of cross-cutting policies depend on both: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each lives in the seam between the two layers and is currently solved by a one-off patch into one neighbor or the other. We argue this seam is best addressed by an architectural change rather than point fixes: insert a third tier, an agent runtime layer, between the framework and the engine, exposing four primitives (observe, score, predict, act) into which any agent-aware policy plugs, with agent identity as the shared coordinate. We map nine concrete policies onto the layer and validate the abstraction in depth on the one with the largest immediate serving-cost lever: KV caching across sessions, instantiated as CacheSage, which learns the per-workload agent transition matrix online and uses it for survival-based eviction and between-step prefetch. Preliminary results on five real multi-agent workloads show +13 to +37 pp cache hit-rate lift, 12% to 29% lower mean TTFT, and 6% to 14% higher throughput over an unmodified serving stack.

Related papers

• Learning Agent Execution for KV-Cache Management in Agentic Serving [25.021363822677515]
  We present CacheScout, an agent-aware KV-cache runtime layer for multi-agent LLM serving.<n>The key insight is that future KV-cache reuse is governed by agent execution semantics rather than cache recency alone.<n>We show that CacheScout improves KV-cache hit rate by 10-18 percentage points, reduces mean TTFT by 18-45%, lowers mean per-turn latency by 29-38%, and increases peak throughput by up to 57%.
  arXiv  Detail & Related papers  (2026-07-16T22:28:19Z)
• Route, Communicate, and Reason: Gated Routing and Adaptive Depth for Efficient Multi-Agent Reasoning [20.207338820226862]
  GRADE is a hierarchical multi-agent system in which four lightweight learned gates jointly govern agent selection, hierarchy depth, inter-agent communication, and branch pruning.<n>At $sim$17B average active parameters, GRADE outperforms all baselines on GSM8K, MMLUPro, and GPQA.<n>On AIME-2025, where model depth dominates, GRADE remains competitive to existing frameworks.
  arXiv  Detail & Related papers  (2026-07-12T16:58:21Z)
• Multi-Agent Computer Use [72.79887808312706]
  We argue that we should move towards evaluating and building multi-agent computer use (MACU) systems.<n>We propose a general multi-agent setup in which a manager model decomposes computer use tasks as a directed acyclic graph (DAG)<n>At each iteration, the manager dispatches parallel CUA subagents to carry out nodes on the ready frontier of the DAG.
  arXiv  Detail & Related papers  (2026-06-01T01:29:36Z)
• Multi-agent Collaboration with State Management [21.19345862774123]
  When multiple agents edit a shared concurrently, their changes can silently conflict and inconsistent views lead to integration failures.<n>Existing multi-agent systems address this through workspace isolation (e.g., one git worktree per agent)<n>We propose STORM, i.e., STate-ORiented Management for multi-agent collaboration.
  arXiv  Detail & Related papers  (2026-05-19T23:45:33Z)
• Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks? [3.636948650519796]
  Small language model (SLM) can achieve comparable performance to frontier models in the task of agentic terminal execution.<n>We present Terminus-4B, which is a post-trained Qwen3-4B model via Supervised Finetuning (SFT) and Reinforcement Learning (RL)<n>We find that Terminus-4B is able to reduce the token usage of the main agent by up to 30% compared to the No Subagent baseline.
  arXiv  Detail & Related papers  (2026-05-04T22:24:24Z)
• AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agents [3.275342475988451]
  Embodied AI agents increasingly rely on large language models (LLMs) for planning, yet per-step LLM calls impose severe latency and cost.<n>We introduce AgenticCache, a framework that reuses cached plans to avoid per-step LLM calls.<n>AgenticCache improves task success rate by 22% on average across 12 configurations, reduces simulation latency by 65%, and lowers token usage by 50%.
  arXiv  Detail & Related papers  (2026-04-27T04:51:15Z)
• OrgAgent: Organize Your Multi-Agent System like a Company [75.47076168155817]
  We introduce OrgAgent, a company-style hierarchical multi-agent framework.<n>We find that multi-agent systems organized in a company-style hierarchy generally outperform other organizational structures.
  arXiv  Detail & Related papers  (2026-04-01T15:21:14Z)
• AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent [57.10083973844841]
  AgentArk is a novel framework to distill multi-agent dynamics into the weights of a single model.<n>We investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios.<n>By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents.
  arXiv  Detail & Related papers  (2026-02-03T19:18:28Z)
• Anemoi: A Semi-Centralized Multi-agent System Based on Agent-to-Agent Communication MCP server from Coral Protocol [0.7288110873705294]
  Anemoi is a semi-centralized MAS built on the Agent-to-Agent (A2A) communication MCP server from Coral Protocol.<n>Unlike traditional designs, Anemoi enables structured and direct inter-agent collaboration, allowing all agents to monitor progress, assess results, identify bottlenecks, and propose refinements in real time.
  arXiv  Detail & Related papers  (2025-08-23T15:45:10Z)
• PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC [98.82146219495792]
  In this paper, we propose a hierarchical agent framework named PC-Agent.<n>From the perception perspective, we devise an Active Perception Module (APM) to overcome the inadequate abilities of current MLLMs in perceiving screenshot content.<n>From the decision-making perspective, to handle complex user instructions and interdependent subtasks more effectively, we propose a hierarchical multi-agent collaboration architecture.
  arXiv  Detail & Related papers  (2025-02-20T05:41:55Z)
• AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents [52.13695464678006]
  This study enhances an LLM-based web agent by simply refining its observation and action space.<n>AgentOccam surpasses the previous state-of-the-art and concurrent work by 9.8 (+29.4%) and 5.9 (+15.8%) absolute points respectively.
  arXiv  Detail & Related papers  (2024-10-17T17:50:38Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• MACRPO: Multi-Agent Cooperative Recurrent Policy Optimization [17.825845543579195]
  We propose a new multi-agent actor-critic method called textitMulti-Agent Cooperative Recurrent Proximal Policy Optimization (MACRPO) We use a recurrent layer in critic's network architecture and propose a new framework to use a meta-trajectory to train the recurrent layer. We evaluate our algorithm on three challenging multi-agent environments with continuous and discrete action spaces.
  arXiv  Detail & Related papers  (2021-09-02T12:43:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.