AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

• URL: http://arxiv.org/abs/2606.09613v1
• Date: Mon, 08 Jun 2026 15:20:23 GMT
• Title: AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
• Abstract summary: We present AGENTSERVESIM, a hardware-aware simulator for multi-turn LLM agent serving.<n>We show that AGENTSERVESIM reproduces real-system behavior within 6% error across key performance metrics while running entirely on commodity CPUs.
• Score: 16.735743806437487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution. Serving these workloads requires scheduling, KV-cache management, and routing policies that use program-level context, including turn dependencies, tool-induced gaps, and reusable KV state. Evaluating such policies directly on real systems is costly, since each design point may require dedicated accelerator time across arrival rates, model scales, serving-instance counts, and memory hierarchies. Simulation offers a scalable alternative, but existing LLM serving simulators target stateless request-level workloads and therefore omit the core dynamics of agent serving: multi-turn program execution, cross-turn cache locality, and KV-cache residency during tool gaps. We present AGENTSERVESIM, a hardware-aware simulator for multi-turn LLM agent serving. AGENTSERVESIM evaluates serving policies at program granularity through composable modules: a Program Orchestrator preserves program identity and turn order, a Tool Simulator materializes tool-induced gaps, a Session-Aware Router maintains program-to-instance affinity for cache-aware dispatch, and a KV Residency Model tracks policy-defined KV placement across HBM, host DRAM/CXL, and eviction. Across real serving deployments and hardware configurations, AGENTSERVESIM reproduces real-system behavior within 6% error across key performance metrics while running entirely on commodity CPUs. These results show that AGENTSERVESIM enables controlled, repeatable exploration of agent-serving policies without requiring exhaustive deployment on costly accelerators.

Related papers

• Characterizing Software Aging in GPU-Based LLM Serving Systems [3.2117529539472716]
  This paper proposes an empirical methodology to study software aging in GPU-based LLM serving systems.<n>We run a 216-hour campaign across six co-located deployments under identical stress conditions.<n>Our results reveal statistically significant memory aging in all deployments, with leak rates strongly dependent on the serving and deployment configuration.
  arXiv  Detail & Related papers  (2026-06-10T10:48:50Z)
• $E^3$-Agent: An Executable and Evolving Agent for Resource Management of Edge Generative Inference [39.00570828094742]
  $E3$-Agent is an evolving agent for edge artificial intelligence generated content (AIGC) resource management.<n>Agent learns online from execution feedback and continuously adapts to unknown and time-varying service-time mappings.
  arXiv  Detail & Related papers  (2026-05-21T12:32:43Z)
• Beyond Short-Horizon: VQ-Memory for Robust Long-Horizon Manipulation in Non-Markovian Simulation Benchmarks [96.60530830276281]
  RuleSafe is a new articulated manipulation benchmark built upon a scalable LLM-aided simulation framework.<n>VQ-Memory is a compact and structured temporal representation that uses vector-quantized variational autoencoders.
  arXiv  Detail & Related papers  (2026-03-10T11:13:54Z)
• RepoLaunch: Automating Build&Test Pipeline of Code Repositories on ANY Language and ANY Platform [49.43594274832262]
  We introduce RepoLaunch, the first agent capable of automatically resolving dependencies, compiling source code, and extracting test results for repositories across arbitrary programming languages and operating systems.<n>RepoLaunch automates the remaining steps, enabling scalable benchmarking and training of coding agents and LLMs.
  arXiv  Detail & Related papers  (2026-03-05T10:15:13Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management [11.653587206247053]
  LLM-based simulations are increasingly adopted across application domains, but remain difficult to scale due to GPU memory pressure.<n>We present ScaleSim, a memory-efficient LLM serving system for large-scale multi-agent simulations.
  arXiv  Detail & Related papers  (2026-01-29T09:52:16Z)
• Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents [12.884297990127985]
  Astraea is a service engine designed to shift the optimization from local segments to the global request lifecycle.<n>It employs a state-aware, hierarchical scheduling algorithm that integrates a request's historical state with future predictions.<n>Astraea reduces average JCT by up to 25.5% compared to baseline methods.
  arXiv  Detail & Related papers  (2025-12-16T06:55:10Z)
• An LLVM-Based Optimization Pipeline for SPDZ [0.0]
  We implement a proof-of-concept LLVM-based optimization pipeline for the SPDZ protocol.<n>Our front end accepts a subset of C with lightweight privacy annotations and lowers it to LLVM IR.<n>Our back end performs data-flow and control-flow analysis on the optimized IR to drive a non-blocking runtime scheduler.
  arXiv  Detail & Related papers  (2025-12-11T20:53:35Z)
• Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live [30.099614426825834]
  Continuum is a serving system to optimize job completion time for multi-turn agent workloads.<n>By predicting tool call durations in agentic, Continuum selectively pins the KV cache in GPU memory with a time-to-live value based on total turn number.<n>Our evaluation on real-world agentic workloads with Llama-3.1 8B/70B models shows that Continuum significantly improves the average job completion times.
  arXiv  Detail & Related papers  (2025-11-04T03:43:05Z)
• Autellix: An Efficient Serving Engine for LLM Agents as General Programs [59.673243129044465]
  Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs.<n>Existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization.<n>We introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies.
  arXiv  Detail & Related papers  (2025-02-19T18:59:30Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.