Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

• URL: http://arxiv.org/abs/2604.08075v1
• Date: Thu, 09 Apr 2026 10:47:20 GMT
• Title: Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
• Abstract summary: Production vLLM fleets typically provision each instance for the worst-case context length.<n>In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts.<n>We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools.
• Score: 9.457255218406333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Production vLLM fleets typically provision each instance for the worst-case context length, leading to substantial KV-cache over-allocation and under-utilized concurrency. In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts, wasting 4-8$\times$ throughput capacity and triggering reliability issues such as OOM crashes, preemption, and request rejections. We identify a common root cause for these inefficiencies: configuration-traffic mismatch. We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools: a high-throughput short-context pool and a high-capacity long-context pool. Each request is routed based on its estimated total token budget, computed using a per-category bytes-to-token ratio that is learned online via exponential moving average from usage.prompt_tokens feedback, eliminating the need for a tokenizer. We also develop a simple analytical model that predicts fleet-level cost savings from workload characteristics and measured throughput differences, enabling practitioners to estimate benefits prior to deployment. Evaluations on real-world traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M, serving Llama-3-70B on A100 GPUs, show that our approach reduces GPU-hours by 31-42%, corresponding to \$2.86M annual savings at fleet scale, while lowering preemption rates by 5.4$\times$ and improving P99 TTFT by 6%. A case study with Qwen3-235B-A22B on AMD MI300X at 10,000 req/s projects \$15.4M in annual savings. The method incurs only O(1) dispatch overhead, adapts automatically to heterogeneous workloads, and composes seamlessly with existing optimizations such as PagedAttention, continuous batching, and prefill-decode disaggregation.

Related papers

• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Lodestar: An Online-Learning LLM Inference Router [6.819928135489163]
  Large language model (LLM) inference tasks are crucial for user-perceived latency.<n>Lodstar is a novel learning-based request routing system for distributed GPU clusters.<n>Lodstar achieves 1.41x lower average TTFT and 1.47x lower P99 TTFT on average.
  arXiv  Detail & Related papers  (2026-05-31T01:31:02Z)
• GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads [6.400220635933459]
  OpenClaw requests are dominated by long, tool-augmented prefixes, including system prompts, conversation history, and tool outputs fed back into the context window.<n>For this workload, with about 28k-30k input tokens and 500 output tokens per request, serving quality is governed by throughput, TTFT, and tail latency rather than short-prompt throughput alone.
  arXiv  Detail & Related papers  (2026-04-27T11:12:33Z)
• A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Model [112.9420001646428]
  Vision-Language-Action (VLA) models have emerged as a powerful paradigm for open-world robot manipulation, but their practical deployment is often constrained by cost.<n>We present A1, a fully open-source and transparent VLA framework designed for low-cost, high- throughput inference.<n>A1 achieves state-of-the-art success rates while significantly reducing inference cost.
  arXiv  Detail & Related papers  (2026-04-07T10:18:40Z)
• Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints [18.52522897906341]
  We study the problem of routing queries to large language models (LLMs)<n>We propose a batch-level, resource-aware routing framework that jointly optimize assignment for each batch.<n> Experiments on two multi-task benchmarks show that improves accuracy by 1-14% on non-robust counterparts.
  arXiv  Detail & Related papers  (2026-03-25T22:24:11Z)
• Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference [12.135271159221178]
  Production vLLM fleets provision every instance for worst-case context.<n>Production vLLM pools dispatch it to one of two vLLM pools.<n> token-budget routing reduces GPU instances by 17-39%.
  arXiv  Detail & Related papers  (2026-03-13T13:06:50Z)
• Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity [5.223181756178433]
  We introduce FastForward, a predictive sparsity framework that accelerates large language model (LLM) prefill through block-wise, context-aware FFN sparsity.<n>FastForward delivers up to 1.45$times$ compute-bound speedup at 50% FFN sparsity with $$ 6% accuracy loss compared to the dense baseline on LongBench.
  arXiv  Detail & Related papers  (2026-01-30T23:23:40Z)
• Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective [31.67506313325633]
  KV caching is a technique for accelerating Large Language Model (LLM) inference by reusing key-value ( KV) pairs from previous queries.<n>The default Least Recently Used (LRU) eviction algorithm struggles with dynamic online query arrivals.<n>We give the first unified mathematical model that captures the core trade-offs between KV cache eviction and query routing.
  arXiv  Detail & Related papers  (2026-01-26T22:20:59Z)
• ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models [4.951427498576812]
  We introduce a novel context compression pipeline called Intermediate Layer Retrieval (ILRe)<n>ILRe encodes context by streaming chunked prefill only up to that layer, and recalls tokens by the attention scores between the input query and full key cache in that specified layer.<n>Without additional post training or operator development, ILRe can process a single $1M$ tokens request in less than half a minute.
  arXiv  Detail & Related papers  (2025-08-25T10:59:02Z)
• Cost-Aware Contrastive Routing for LLMs [57.30288453580456]
  We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space.<n>CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%.
  arXiv  Detail & Related papers  (2025-08-17T20:16:44Z)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression [52.0792918455501]
  We propose a novel two-stage policy optimization framework that directly approximates the optimal advantage function.<n>$A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks.<n>It reduces training time by up to 2$times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL.
  arXiv  Detail & Related papers  (2025-05-27T03:58:50Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.