Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

• URL: http://arxiv.org/abs/2609.16206v1
• Date: Mon, 14 Sep 2026 18:39:18 GMT
• Title: Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
• Abstract summary: We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class.<n>Across three mixed, bursty arrival traces, the router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class. We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation. Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic. It also shows the lowest variance across traces. It beats round robin and the length heuristic on all three traces and least loaded on two. On the third, it trails by 0.003, within run to run noise. Hardware calibration matters: simulator derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.

Related papers

• LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving [12.697860184583483]
  Co-batching improves throughput but obscures per-tenant usage and limits control.<n>We present LLMVisor, a roofline-guided latency model that captures the memory-bound and compute-bound phases.<n>Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x.
  arXiv  Detail & Related papers  (2026-08-09T00:28:05Z)
• Think Before You Grid-Search: Floor-First Triage for LLM Serving [0.0]
  LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed.<n>We argue for the reverse discipline: estimation is the analytical layer of profiling.
  arXiv  Detail & Related papers  (2026-07-07T06:11:54Z)
• GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices [3.3723515662362265]
  We present GPUSparse, a system for GPU-accelerated exact learned sparse retrieval.<n>A GPU-parallel inverted index with block-aligned, warp-coalesced posting lists is used.<n>Batched scatter-add scoring algorithm processes hundreds of queries simultaneously.
  arXiv  Detail & Related papers  (2026-06-24T23:07:57Z)
• AdpSplit: Error-Driven Adaptive Splitting for Faster Geometry Discovery in 3D Gaussian Splatting [61.977804325410695]
  We introduce AdpSplit, an error-driven adaptive split operator that determines the number of split children and initializes the child parameters from L1-pixel-error region statistics.<n>Across the MipNeRF360, Deep-Blending, and Tanks&Temples datasets, AdpSplit reduces the training time of multiple accelerated 3DGS pipelines by 9.2%-22.3%.<n>With FastGS, AdpSplit matches the full-schedule PSNR on MipNeRF360 while reducing training time by 16.4%, corresponding to a 12.6x acceleration over vanilla 3DGS.
  arXiv  Detail & Related papers  (2026-05-07T19:23:16Z)
• Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving [9.457255218406333]
  Production vLLM fleets typically provision each instance for the worst-case context length.<n>In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts.<n>We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools.
  arXiv  Detail & Related papers  (2026-04-09T10:47:20Z)
• Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference [12.135271159221178]
  Production vLLM fleets provision every instance for worst-case context.<n>Production vLLM pools dispatch it to one of two vLLM pools.<n> token-budget routing reduces GPU instances by 17-39%.
  arXiv  Detail & Related papers  (2026-03-13T13:06:50Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity [52.88892280536302]
  We introduce SparseLoRA, a method that accelerates fine-tuning through contextual sparsity.<n>We show that SparseLoRA reduces computational cost by up to 2.2 times and a measured speedup of up to 1.6 times.
  arXiv  Detail & Related papers  (2025-06-19T17:53:34Z)
• KurTail : Kurtosis-based LLM Quantization [51.24081396305435]
  KurTail is a new post-training quantization scheme that mitigates outliers in the activations of large language models.<n>It offers a 13.3% boost in MMLU accuracy and a 15.5% drop in Wiki perplexity compared to QuaRot.<n>It also outperforms SpinQuant with a 2.6% MMLU gain and reduces perplexity by 2.9%, all while reducing the training cost.
  arXiv  Detail & Related papers  (2025-03-03T12:43:06Z)
• Efficient Offline Policy Optimization with a Learned Model [83.64779942889916]
  MuZero Unplugged presents a promising approach for offline policy learning from logged data. It conducts Monte-Carlo Tree Search (MCTS) with a learned model and leverages Reanalyze algorithm to learn purely from offline data. This paper investigates a few hypotheses where MuZero Unplugged may not work well under the offline settings.
  arXiv  Detail & Related papers  (2022-10-12T07:41:04Z)
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers [65.60007071024629]
  We show that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy. We show experimentally that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy.
  arXiv  Detail & Related papers  (2021-10-13T20:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.