Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems

• URL: http://arxiv.org/abs/2608.24650v2
• Date: Wed, 26 Aug 2026 03:57:59 GMT
• Title: Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems
• Abstract summary: We present Simthesizer, a framework that realizes agent-driven simulator development.<n>Simthesizer introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow.<n>On identical workloads, Simthesizer simulates up to 284.96x and 23.19x faster than two state-of-the-art simulators.
• Score: 10.265484416904352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: System-level simulation is an essential tool for exploring the rapidly expanding design space of LLM serving systems, where real deployments remain costly and often infeasible. However, modern LLM serving now evolves faster than human-driven simulator development can track, and emerging workloads and mechanisms, from agentic workflows to disaggregated serving, no longer fit the monolithic simulation pipeline that existing simulators assume. Each new mechanism therefore demands an invasive rewrite, leaving a widening development gap between deployed serving systems and the simulators that model them. To close this gap, we present Simthesizer, a framework that realizes agent-driven simulator development. Simthesizer introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow, including the control decisions that coordinate it, and realizes it as a unified dynamic graph in Simthesizer simulator. Synthesizer agent, a harnessed coding agent, then lowers natural-language feature requests onto this abstraction under simulator-specific guardrails and fidelity validation, evolving one shared simulator instead of building a new one for every feature. Under the same coding agent and harnesses, extensions built on Simthesizer follow a vLLM-based real system with 2.51% average throughput error, versus 6.03% for extensions built on existing simulators. On identical workloads, Simthesizer also simulates up to 284.96x and 23.19x faster than two state-of-the-art simulators, LLMServingSim2.0 and Vidur, respectively.

Related papers

• GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments [63.222920701596564]
  Training embodied agents in the real world requires skilled operators and expensive hardware.<n> Simulation environments offer a compelling alternative by enabling large-scale, cost-effective data augmentation.<n>Fastly constructing high-fidelity simulation scenes with a minimal sim-to-real gap has become a critical objective in robot learning.
  arXiv  Detail & Related papers  (2026-06-16T05:00:42Z)
• IR-SIM: A Lightweight Skill-Native Simulator for Navigation, Learning, and Benchmarking [59.6657632820611]
  IR-SIM is a skill-native navigation simulator designed for rapid scenario construction, benchmarking, and robot learning.<n> scenarios are entirely defined by YAML configuration files that specify mobile robot kinematics, geometric collision checking, LiDAR sensing, visualization, and behavior modules.<n>The experiments showcase the convenience and versatility of IR-SIM in multiple tasks.
  arXiv  Detail & Related papers  (2026-06-07T16:55:37Z)
• LychSim: A Controllable and Interactive Simulation Framework for Vision Research [13.971578506119647]
  We present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5.<n>LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs.
  arXiv  Detail & Related papers  (2026-05-12T17:40:38Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure [4.1898448424363695]
  Large language model (LLM) serving infrastructures are undergoing a shift toward heterogeneous and disaggregation.<n>This paper presents LLMServingSim 2.0, a unified system-level simulator designed to make runtime-driven hardware-software interactions explicit and analyzable.
  arXiv  Detail & Related papers  (2026-02-26T14:22:17Z)
• SimuAgent: An LLM-Based Simulink Modeling Assistant Enhanced with Reinforcement Learning [3.1436750864792375]
  We introduce SimuAgent, an LLM-powered modeling and simulation agent tailored for Simulink.<n>SimuAgent replaces XML with a concise, dictionary-style Python representation, dramatically cutting token counts.<n>A lightweight plan-execute architecture, trained in two stages, equips the agent with both low-level tool skills and high-level design reasoning.
  arXiv  Detail & Related papers  (2026-01-08T18:10:35Z)
• Simulating Environments with Reasoning Models for Agent Training [55.98861707136674]
  Building bespoke environments for training is heavy, brittle, and limits progress.<n>We propose two frameworks: Simia-SFT and Simia-RL.<n>Simia-SFT and Simia-RL enable scalable agent training without environment engineering.
  arXiv  Detail & Related papers  (2025-11-03T18:29:57Z)
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training [55.72784274656801]
  We introduce a scalable paradigm that generates structured UI states and transitions to synthesize training trajectories at scale.<n>Our paradigm integrates a digital world simulator for diverse UI states, a guided rollout process for coherent exploration, and a trajectory wrapper.<n>Experiments on WebArena and AndroidWorld show that UI-Simulator rivals or surpasses open-source agents trained on real UIs with significantly better robustness.
  arXiv  Detail & Related papers  (2025-10-16T17:59:38Z)
• G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration [48.948187359727996]
  G-Sim is a hybrid framework that automates simulator construction with rigorous empirical calibration.<n>It produces reliable, causally-informed simulators, mitigating data-inefficiency and enabling robust system-level interventions.
  arXiv  Detail & Related papers  (2025-06-10T22:14:34Z)
• Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimation [13.326000659635378]
  Phantora is a hybrid GPU cluster simulator for performance estimation of machine learning training workloads.<n>It allows direct reuse of ML framework source code in simulation, avoiding the need for reimplementation.<n>Phantora supports three state-of-the-art training frameworks out-of-the-box.
  arXiv  Detail & Related papers  (2025-05-02T22:36:24Z)
• LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale [17.00936774784349]
  There is a lack of simulation infrastructure capable of accurately modeling versatile hardware-software behaviors in large language model (LLM) serving systems. This paper aims to develop an effective simulation tool, called LLMServingSim, to support future research in LLM serving systems.
  arXiv  Detail & Related papers  (2024-08-10T09:26:15Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.