HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

• URL: http://arxiv.org/abs/2607.22578v1
• Date: Sun, 07 Jun 2026 03:14:57 GMT
• Title: HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
• Abstract summary: HeraSys is a serving system designed to optimize the end-to-end performance of concurrent processing.<n>HeraSys reduces P99 latency by up to 2.17$timestimes serving throughput by up to 1.85timestimes under strict latency guarantees.
• Score: 11.533835891575565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The proliferation of Large Language Models (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows. However, existing solutions typically prioritize intra-workflow optimization, largely neglecting the significant potential for inter-workflow optimization. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end-to-end performance of concurrent workflows. Through fine-grained orchestration, HeraSys eliminates cross-workflow computational redundancy via structural node merging and reuse. Furthermore, HeraSys introduces a load-aware joint scheduling policy that dynamically manages execution order by evaluating both inter- and intra-query priorities. By integrating a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, thereby substantially improving system throughput. Extensive experiments demonstrate that HeraSys reduces P99 latency by up to 2.17$\times$ and increases serving throughput by up to 1.85$\times$ under strict latency guarantees.

Related papers

• Multi-Timescale Latent-Action DRL for Joint Optimization in Edge-Cloud Networks [48.421359451414155]
  Load imbalance across edge cloud and layers degrades latency performance in.<n> hierarchical edge-cloud computing systems under dynamic task arrivals and heterogeneous resources.<n>We propose a two-time deep learning study with an action space (2MDRL-LA) to jointly optimize service placement.<n>It achieves a 20.8% reduction in average e2e latency and a 13% improvement in resource utilization over the scheme.
  arXiv  Detail & Related papers  (2026-06-30T11:56:15Z)
• CoMem: Context Management with A Decoupled Long-Context Model [68.6597154502361]
  CoMem is a framework that decouples memory management from the primary agent workflow.<n>We show that CoMem provides 1.4x latency improvements upon vanilla long-context solutions.
  arXiv  Detail & Related papers  (2026-05-29T04:59:32Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving [13.856291757420012]
  Long-running requests monopolize resources and delay higher-priority ones, leading to widespread time-to-first-token (TTFT) service level violations.<n>We propose FlowPrefill, a TTFT-goodput-optimized serving system that balances execution granularity against scheduling overheads.<n>We show that FlowPrefill improves maximum goodput by up to 5.6$times$ compared to state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-02-18T16:57:45Z)
• Cloud-Fog-Edge Collaborative Computing for Sequential MIoT Workflow: A Two-Tier DDPG-Based Scheduling Framework [2.5263430338308557]
  Medical Internet of Things (MIoT) demands stringent end-to-end latency guarantees for sequential healthcare deployed over heterogeneous cloud-fog-edge infrastructures.<n>We propose a Two-tier DDPG-based scheduling framework that decomposes the scheduling decision into a hierarchical process.<n> Experiments results validate our approach, demonstrating increasingly superior performance over baselines as complexity rises.
  arXiv  Detail & Related papers  (2025-10-24T03:58:31Z)
• AoI-Aware Task Offloading and Transmission Optimization for Industrial IoT Networks: A Branching Deep Reinforcement Learning Approach [43.261887758877386]
  In the Industrial Internet of Things (IIoT), the frequent transmission of large amounts of data over wireless networks should meet the stringent timeliness requirements.<n>We propose an age-of-information (AoI)-aware multi-base station (BS) real-time monitoring framework to support extensive IIoT deployments.
  arXiv  Detail & Related papers  (2025-10-18T09:14:39Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers [58.5711048151424]
  We introduce SPARSEK Attention, a novel sparse attention mechanism designed to overcome computational and memory obstacles. Our approach integrates a scoring network and a differentiable top-k mask operator, SPARSEK, to select a constant number of KV pairs for each query. Experimental results reveal that SPARSEK Attention outperforms previous sparse attention methods.
  arXiv  Detail & Related papers  (2024-06-24T15:55:59Z)
• Client Orchestration and Cost-Efficient Joint Optimization for NOMA-Enabled Hierarchical Federated Learning [55.49099125128281]
  We propose a non-orthogonal multiple access (NOMA) enabled HFL system under semi-synchronous cloud model aggregation. We show that the proposed scheme outperforms the considered benchmarks regarding HFL performance improvement and total cost reduction.
  arXiv  Detail & Related papers  (2023-11-03T13:34:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.