TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving

• URL: http://arxiv.org/abs/2608.25523v1
• Date: Wed, 26 Aug 2026 08:33:10 GMT
• Title: TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
• Abstract summary: We propose TOPAS, a Task-Oriented Prefix-Aware Scheduler.<n> TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path.<n>Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%.
• Score: 8.123227365399869
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance, we here propose TOPAS, a Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution. TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption. A task-level aging mechanism is also incorporated to prevent starvation. We implement TOPAS within the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on the synthetic workloads, while lowering mean JCT by 9.8% on MetaGPT-SOP and mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.

Related papers

• Cloud Workflow Scheduling Based on Graph Attention-Driven Hierarchical Reinforcement Learning [10.978371228753518]
  We develop a Graph Attention-Driven Reinforcement Learning (GA-HRL) framework and model scheduling process.<n>It balances deadline satisfaction, container utilization, and energy consumption while dealing with task-execution speeds, placement-dependent communication, and coupled task and container decisions.<n>Experiments on the 2018 Alibaba cluster trace show that GA-HRL maintains competitive workflow success rate and, in settings where success is comparable, achieves higher container utilization and lower energy consumption.
  arXiv  Detail & Related papers  (2026-09-14T02:50:40Z)
• Workload-Aware Caching for Multi-Agent Systems [3.2309496907010384]
  Cache eviction policies treat all cached entries uniformly based on access history, ignoring structural and workload signals.<n>We present a workload-aware eviction policy that combines three signals, namely recomputation cost, DAG dependency count, and agent invocation frequency, into a unified scoring function.<n>Our policy reduces latency by up to 64.7% relative to the uncached baseline and achieves on average a 31.1% latency reduction over the next best finite-capacity baseline.
  arXiv  Detail & Related papers  (2026-06-14T02:13:27Z)
• HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization [11.533835891575565]
  HeraSys is a serving system designed to optimize the end-to-end performance of concurrent processing.<n>HeraSys reduces P99 latency by up to 2.17$timestimes serving throughput by up to 1.85timestimes under strict latency guarantees.
  arXiv  Detail & Related papers  (2026-06-07T03:14:57Z)
• SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters [13.891522069967507]
  We propose a shift to program-level scheduling for compound AI workloads.<n>We present SAGA, a distributed scheduler that implements this abstraction through three mechanisms.
  arXiv  Detail & Related papers  (2026-05-01T09:05:28Z)
• Why Search When You Can Transfer? Amortized Agentic Workflow Design from Structural Priors [51.95306551061392]
  Automated agentic workflow design currently relies on per-task iterative search.<n>We present SWIFT, a framework that amortizes workflow design into reusable structural priors.<n> SWIFT outperforms the state-of-the-art search-based method on five benchmarks.
  arXiv  Detail & Related papers  (2026-04-27T21:25:00Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents [12.884297990127985]
  Astraea is a service engine designed to shift the optimization from local segments to the global request lifecycle.<n>It employs a state-aware, hierarchical scheduling algorithm that integrates a request's historical state with future predictions.<n>Astraea reduces average JCT by up to 25.5% compared to baseline methods.
  arXiv  Detail & Related papers  (2025-12-16T06:55:10Z)
• DEPO: Dual-Efficiency Preference Optimization for LLM Agents [75.6723341304463]
  We propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps.<n>Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance.
  arXiv  Detail & Related papers  (2025-11-19T12:38:43Z)
• Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference [4.7730970530715835]
  Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging.<n>We introduce QLLM, a novel inference system designed for Mixture of Experts (MoE) models.<n>QLLM enables expert-level preemption, deferring BE job execution while minimizing LS time-to-first-token (TTFT)
  arXiv  Detail & Related papers  (2025-03-12T11:56:01Z)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging [80.17238673443127]
  LiNeS is a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance.<n>LiNeS demonstrates significant improvements in both single-task and multi-task settings across various benchmarks in vision and natural language processing.
  arXiv  Detail & Related papers  (2024-10-22T16:26:05Z)
• DASA: Delay-Adaptive Multi-Agent Stochastic Approximation [64.32538247395627]
  We consider a setting in which $N$ agents aim to speedup a common Approximation problem by acting in parallel and communicating with a central server. To mitigate the effect of delays and stragglers, we propose textttDASA, a Delay-Adaptive algorithm for multi-agent Approximation.
  arXiv  Detail & Related papers  (2024-03-25T22:49:56Z)
• A Feedback Scheme to Reorder a Multi-Agent Execution Schedule by Persistently Optimizing a Switchable Action Dependency Graph [65.70656676650391]
  We consider multiple Automated Guided Vehicles (AGVs) navigating a common workspace to fulfill various intralogistics tasks. One approach is to construct an Action Dependency Graph (ADG) which encodes the ordering of AGVs as they proceed along their routes. If the workspace is shared by dynamic obstacles such as humans or third party robots, AGVs can experience large delays. We present an online method to repeatedly modify acyclic ADG to minimize route completion times of each AGV.
  arXiv  Detail & Related papers  (2020-10-11T14:39:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.