Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving

• URL: http://arxiv.org/abs/2608.15762v1
• Date: Sun, 16 Aug 2026 14:31:56 GMT
• Title: Global Simulation-Guided Dynamic Operator Scheduling for Efficient Multi-Tenant Model Serving
• Abstract summary: We present SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving.<n>The key idea is to expose cluster-wide operator execution state and enable what-if reasoning over scheduling decisions.<n>We show that SliceScheduler improves token throughput by 1.10--2.29$times$ compared to existing approaches.
• Score: 13.853492308825052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Container-granularity scheduling leaves abundant short-lived idle slices within containers unexploited. Reallocating containers is too heavyweight to utilize such fine-grained opportunities under SLA constraints, and operator-level scheduling requires reasoning about dependencies, memory safety, and cluster-wide execution dynamics in real time. In this paper, we present SliceScheduler, a dynamic operator-level scheduling system for multi-tenant model serving. The key idea is to expose cluster-wide operator execution state and enable what-if reasoning over scheduling decisions. SliceScheduler consists of four key components. First, we introduce the Global Mapping Graph (GMG), a unified abstraction that captures operator dependencies, tensor shapes, resource mappings, and execution states, providing a real-time, cluster-wide view with explicit resource semantics. Second, we build a global simulator on top of GMG to predict operator-level execution and memory evolution under candidate placements. Third, we design an incremental, simulation-based scheduling module that selects placements to exploit fragmented idle slices while avoiding memory violations and preserving SLA. Finally, we develop an operator executor that materializes scheduling decisions on GPUs and coordinates computation and cross-accelerator transfers. We implement SliceScheduler as a PyTorch backend and evaluate it using production trace replay. Experimental results show that SliceScheduler improves token throughput by 1.10--2.29$\times$ compared to existing approaches, while maintaining SLA violations within 9\%. SliceScheduler demonstrates that operator-level scheduling is a practical and effective approach to improving GPU utilization for multi-tenant LLM serving.

Related papers

• Cloud Workflow Scheduling Based on Graph Attention-Driven Hierarchical Reinforcement Learning [10.978371228753518]
  We develop a Graph Attention-Driven Reinforcement Learning (GA-HRL) framework and model scheduling process.<n>It balances deadline satisfaction, container utilization, and energy consumption while dealing with task-execution speeds, placement-dependent communication, and coupled task and container decisions.<n>Experiments on the 2018 Alibaba cluster trace show that GA-HRL maintains competitive workflow success rate and, in settings where success is comparable, achieves higher container utilization and lower energy consumption.
  arXiv  Detail & Related papers  (2026-09-14T02:50:40Z)
• PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents [79.28372522980153]
  PeakBench is a benchmark of executable multi-tool dependency annotations and measured resource profiles.<n>We show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints.<n>PeakBench is a useful testbed for diagnosing resource-aware agent behavior.
  arXiv  Detail & Related papers  (2026-08-25T12:56:08Z)
• Kalypso: Relational LLM Serving [6.264258565843104]
  Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan.<n>We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm.<n>Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads.
  arXiv  Detail & Related papers  (2026-07-26T19:29:31Z)
• PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR [33.08614139144317]
  We present PlexRL, a cluster-level runtime for multiplexing unified LLM services across RLVR jobs.<n>Our implementation and evaluations demonstrate that PlexRL significantly improves effective cluster capacity and reduces user GPU hour cost by maximum 37.58%.
  arXiv  Detail & Related papers  (2026-05-20T07:55:06Z)
• Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines [0.7551404527457958]
  We describe a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic onto a GPU cluster.<n>Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently.
  arXiv  Detail & Related papers  (2026-04-16T16:15:29Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• Semantic-Aware Scheduling for GPU Clusters with Large Language Models [60.14838697778884]
  We propose SchedMate, a framework that bridges the semantic gap between schedulers and jobs they manage.<n>SchedMate extracts deep insights from overlooked, unstructured data sources: source code, runtime logs, and historical jobs.<n>We show SchedMate reduces average job completion times by up to 1.91x, substantially enhancing the scheduling performance.
  arXiv  Detail & Related papers  (2025-10-02T02:01:02Z)
• Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs [3.7758841366694353]
  We survey scheduling techniques from the literature and from practical serving systems.<n>We find that schedulers from the literature often achieve good performance but introduce significant complexity.<n>In contrast, schedulers in practical deployments often leave easy performance gains on the table but are easy to implement, deploy and configure.
  arXiv  Detail & Related papers  (2024-10-23T13:05:46Z)
• Tree-Planner: Efficient Close-loop Task Planning with Large Language Models [63.06270302774049]
  Tree-Planner reframes task planning with Large Language Models into three distinct phases. Tree-Planner achieves state-of-the-art performance while maintaining high efficiency.
  arXiv  Detail & Related papers  (2023-10-12T17:59:50Z)
• NumS: Scalable Array Programming for the Cloud [82.827921577004]
  We present NumS, an array programming library which optimize NumPy-like expressions on task-based distributed systems. This is achieved through a novel scheduler called Load Simulated Hierarchical Scheduling (LSHS) We show that LSHS enhances performance on Ray by decreasing network load by a factor of 2x, requiring 4x less memory, and reducing execution time by 10x on the logistic regression problem.
  arXiv  Detail & Related papers  (2022-06-28T20:13:40Z)
• Gradient Coding with Dynamic Clustering for Straggler-Tolerant Distributed Learning [55.052517095437]
  gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is $straggling$ workers. Coded distributed techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. We propose a novel dynamic GC scheme, which assigns redundant data to workers to acquire the flexibility to choose from among a set of possible codes depending on the past straggling behavior.
  arXiv  Detail & Related papers  (2021-03-01T18:51:29Z)
• Gradient Coding with Dynamic Clustering for Straggler Mitigation [57.9123881133818]
  GC-DC regulates the number of straggling workers in each cluster based on the straggler behavior in the previous iteration. We numerically show that GC-DC provides significant improvements in the average completion time (of each iteration) with no increase in the communication load compared to the original GC scheme.
  arXiv  Detail & Related papers  (2020-11-03T18:52:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.