Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines

• URL: http://arxiv.org/abs/2604.15186v1
• Date: Thu, 16 Apr 2026 16:15:29 GMT
• Title: Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
• Abstract summary: We describe a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic onto a GPU cluster.<n>Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently.
• Score: 0.7551404527457958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic workflows carry out complex tasks by orchestrating multiple large language models (LLMs) and tools. Serving such workflows at a target throughput with low latency is challenging because they can be defined using arbitrary agentic frameworks and exhibit unpredictable execution times: execution may branch, fan-out, or recur in data-dependent ways. Since LLMs in workflows often outnumber available GPUs, their execution also leads to GPU oversubscription. We describe Scepsy, a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic workflows onto a GPU cluster. Scepsy exploits the insight that, while agentic workflows have unpredictable end-to-end latencies, the shares of each LLM's total execution times are comparatively stable across executions. Scepsy decides on GPU allocations based on these aggregate shares: first, it profiles the LLMs under different parallelism degrees. It then uses these statistics to construct an Aggregate LLM Pipeline, which is a lightweight latency/throughput predictor for allocations. To find a GPU allocation that minimizes latency while achieving a target throughput, Scepsy uses the Aggregate LLM Pipeline to explore a search space over fractional GPU shares, tensor parallelism degrees, and replica counts. It uses a hierarchical heuristic to place the best allocation onto the GPU cluster, minimizing fragmentation, while respecting network topology constraints. Our evaluation on realistic agentic workflows shows that Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently or rely on user-specified allocations.

Related papers

• Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving [13.853492308825052]
  We present SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving.<n>The key idea is to expose cluster-wide operator execution state and enable what-if reasoning over scheduling decisions.<n>We show that SliceScheduler improves token throughput by 1.10--2.29$times$ compared to existing approaches.
  arXiv  Detail & Related papers  (2026-08-16T14:31:56Z)
• Kalypso: Relational LLM Serving [6.264258565843104]
  Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan.<n>We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm.<n>Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads.
  arXiv  Detail & Related papers  (2026-07-26T19:29:31Z)
• Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations [53.53864188270018]
  Large language models (LLMs) have advanced recommendation systems (RSs)<n>We propose MARC to explicitly control the modularity of LLMs.<n>MARC achieved a 2.82% eCPM lift in an online A/B test within a large-scale commercial search advertising scenario.
  arXiv  Detail & Related papers  (2026-04-20T12:08:58Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• Data Driven Optimization of GPU efficiency for Distributed LLM Adapter Serving [2.6336040306318274]
  Large Language Model (LLM) adapters enable low-cost model specialization.<n>LLM adapters introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.<n>This paper presents a data-driven pipeline that computes an adapter placement that serves the workload with the minimum number of GPU.
  arXiv  Detail & Related papers  (2026-02-27T14:22:51Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• Performance of Small Language Model Pretraining on FABRIC: An Empirical Study [2.2070336216767763]
  In this work, we investigate the performance of pretraining techniques for smaller-sized LLMs on an experimental testbed available to academic users at no charge.<n>We use GPT-2 medium and large models and pretrained them using open-source packages, namely, Alpa and Ray.<n>We observed that Alpa's execution plans that collectively optimized intra-operator and inter-operator/pipeline parallelism consistently performed the best when GPUs were geographically distributed.
  arXiv  Detail & Related papers  (2026-02-02T17:58:47Z)
• VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG [2.0929459605817193]
  Retrieval-Augmented Generation (RAG) systems combine vector similarity search with large language models (LLMs) to deliver context-aware responses.<n>We present VectorLiteRAG, a deployment-friendly RAG system that achieves latency-compliant inference without requiring additional hardware resources.
  arXiv  Detail & Related papers  (2025-04-11T19:18:41Z)
• Multi-Bin Batching for Increasing LLM Inference Throughput [19.652542432683234]
  Large language models (LL) grow in popularity improving the efficiency of their systems.<n> requests is a critical step in scheduling jobs on servers.<n> requests often have varying generation lengths, causing resource underutilization.<n>We formalize this problem from a queueing-theoretic perspective, and aim to design a throughput control policy.
  arXiv  Detail & Related papers  (2024-12-03T03:16:12Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• An LLM Compiler for Parallel Function Calling [68.04566807806071]
  We introduce LLMCompiler, which executes functions in parallel to efficiently orchestrate multiple function calls. We observe consistent latency speedup of up to 3.7x, cost savings of up to 6.7x, and accuracy improvement of up to 9% compared to ReAct.
  arXiv  Detail & Related papers  (2023-12-07T18:32:04Z)
• NumS: Scalable Array Programming for the Cloud [82.827921577004]
  We present NumS, an array programming library which optimize NumPy-like expressions on task-based distributed systems. This is achieved through a novel scheduler called Load Simulated Hierarchical Scheduling (LSHS) We show that LSHS enhances performance on Ray by decreasing network load by a factor of 2x, requiring 4x less memory, and reducing execution time by 10x on the logistic regression problem.
  arXiv  Detail & Related papers  (2022-06-28T20:13:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.