DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference

• URL: http://arxiv.org/abs/2606.02982v1
• Date: Tue, 02 Jun 2026 00:39:31 GMT
• Title: DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference
• Abstract summary: Large language model (LLM) inference services have increased the demand for efficient multi-tenant GPU scheduling.<n>This paper presents DriftSched, an adaptive-aware scheduling framework for multi-tenant GPU scheduling.<n>The framework evaluates FIFO, Priority, Weighted, Shortest-Job-First (SJF), and Aging Priority scheduling policies under heterogeneous multi-tenant workloads.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inference runtimes such as vLLM improve throughput through continuous batching and optimized memory management, accurately estimating the runtime cost of heterogeneous inference requests remains a significant challenge. In practice, observed output lengths often deviate from admission-time estimates, creating runtime token drift that can lead to workload misclassification, queue imbalance, increased tail latency, and degraded Quality-of-Service (QoS). This paper presents DriftSched, an adaptive QoS-aware scheduling framework for multi-tenant LLM inference serving on NVIDIA L4 GPUs. DriftSched combines workload classification, token-budget estimation, tenant-aware queue management, and runtime feedback-driven drift compensation to improve admission-time scheduling decisions. The framework evaluates FIFO, Priority, Weighted, Shortest-Job-First (SJF), and Aging Priority scheduling policies under heterogeneous multi-tenant workloads. Experimental results demonstrate measurable runtime token drift across workload categories. Adaptive bias correction reduces workload estimation error by an average of 38.8% (MAE) and 40.5% (RMSE), improving workload classification stability and scheduling accuracy. Among all evaluated schedulers, SJF achieves the best overall performance, reducing median end-to-end latency by approximately 42% and P99 latency by approximately 16% relative to FIFO under sustained GPU contention. The work contributes an adaptive drift-aware scheduling architecture, a runtime token-drift compensation mechanism, and a reproducible benchmarking framework for evaluating QoS-aware LLM inference scheduling on shared GPU infrastructure.

Related papers

• SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data [49.370849653460716]
  We propose SOPE, an algorithm that uses an actor-aligned Off-Policy Policy Evaluation signal as an automated early-stopping mechanism.<n> evaluated on 25 continuous control tasks from the Minari benchmark suite.<n>SOPE improves baseline performance by up to 45.6% while reducing the required TFLOPs by up to 22x.
  arXiv  Detail & Related papers  (2026-05-07T08:32:09Z)
• TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs [8.818252253980985]
  TempoNet is a reinforcement learning scheduler that pairs a permutation-invariant Transformer with a deep Q-approximation.<n>A latency-aware sparse attention stack with blockwise top-k selection and locality-sensitive chunking enables global reasoning over unordered task sets.
  arXiv  Detail & Related papers  (2026-02-20T09:56:23Z)
• EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference [1.7969777786551429]
  EWSJF (Effective Workload-based Shortest Job First) learns workload structure in real time to jointly improve fairness and throughput.<n>EWSJF improves end-to-end throughput by over 30% and reduces average Time-To-First-Token for short requests by up to 4x compared to FCFS.
  arXiv  Detail & Related papers  (2026-01-29T14:14:16Z)
• Seer Self-Consistency: Advance Budget Estimation for Adaptive Test-Time Scaling [55.026048429595384]
  Test-time scaling improves the inference performance of Large Language Models (LLMs) but also incurs substantial computational costs.<n>We propose SeerSC, a dynamic self-consistency framework that simultaneously improves token efficiency and latency.
  arXiv  Detail & Related papers  (2025-11-12T13:57:43Z)
• FairBatching: Fairness-Aware Batch Formation for LLM Inference [2.0917668141703207]
  This work identifies the root cause of this unfairness: the non-monotonic nature of Time--Tokens (TBT)<n>We propose Fair the Prioritizing, a novel system that enforces fair resource allocation between fill and decode tasks.
  arXiv  Detail & Related papers  (2025-10-16T07:43:56Z)
• Intra-request branch orchestration for efficient LLM reasoning [52.68946975865865]
  Large Language Models (LLMs) increasingly rely on inference-time reasoning algorithms to improve accuracy on complex tasks.<n>Prior work has largely focused on reducing token usage, often at the expense of accuracy, while overlooking other latency factors.<n>We present DUCHESS, an LLM serving system that reduces cost and latency without sacrificing accuracy through intra-request branch orchestration guided by predictions.
  arXiv  Detail & Related papers  (2025-09-29T15:52:08Z)
• MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment [14.392166280035122]
  Large language models (LLMs) deployed on edge servers are increasingly used in latency-sensitive applications such as personalized assistants, recommendation, and content moderation.<n>Existing retraining strategies either delay model updates, over-commit resources to retraining, or overlook iteration-level retraining granularity.<n>We propose MACE, a hybrid LLM system that colocates concurrent inference (prefill, decode) and fine-tuning, with intelligent memory management to maximize task performance while promising inference throughput.
  arXiv  Detail & Related papers  (2025-09-28T18:45:28Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks [56.37880529653111]
  The demand for large computation model (LAIM) services is driving a paradigm shift from traditional cloud-based inference to edge-based inference for low-latency, privacy-preserving applications.<n>In this paper, we investigate the LAIM-inference scheme, where a pre-trained LAIM is pruned and partitioned into on-device and on-server sub-models for deployment.
  arXiv  Detail & Related papers  (2025-05-14T08:18:55Z)
• Client Orchestration and Cost-Efficient Joint Optimization for NOMA-Enabled Hierarchical Federated Learning [55.49099125128281]
  We propose a non-orthogonal multiple access (NOMA) enabled HFL system under semi-synchronous cloud model aggregation. We show that the proposed scheme outperforms the considered benchmarks regarding HFL performance improvement and total cost reduction.
  arXiv  Detail & Related papers  (2023-11-03T13:34:44Z)
• Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning [26.813145949399427]
  In federated learning (FL), devices upload their local model updates via wireless channels. We propose a joint device scheduling and resource allocation policy to maximize the model accuracy. Experiments show that the proposed policy outperforms state-of-the-art scheduling policies.
  arXiv  Detail & Related papers  (2020-07-14T16:46:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.