Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

• URL: http://arxiv.org/abs/2604.09613v2
• Date: Tue, 14 Apr 2026 18:06:29 GMT
• Title: Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
• Abstract summary: Production vLLM fleets provision every instance for worst-case context.<n>Production vLLM pools dispatch it to one of two vLLM pools.<n> token-budget routing reduces GPU instances by 17-39%.
• Score: 12.135271159221178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Production vLLM fleets provision every instance for worst-case context length, wasting 4-8x concurrency on the 80-95% of requests that are short and simultaneously triggering KV-cache failures -- OOM crashes, preemption storms, and request rejections. Both problems share a single root cause: configuration-traffic mismatch. We propose token-budget-aware pool routing: estimate each request's total token budget using a self-calibrating per-category bytes-per-token ratio, then dispatch it to one of two vLLM pools -- a high-throughput short pool or a high-capacity long pool -- each right-sized for its workload class. The ratio is learned online via exponential moving average from usage.prompt_tokens feedback, requiring no tokenizer. A closed-form cost model, savings = alpha * (1 - 1/rho), predicts fleet-level GPU savings from two observable quantities: the short-traffic fraction alpha and the throughput gain ratio rho. On traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M serving Llama-3-70B on A100 GPUs, token-budget routing reduces GPU instances by 17-39% (\$1.2-2.0M/yr at 1,000 req/s), with savings verified by a self-contained discrete-event simulator. A case study projecting Qwen3-235B-A22B on AMD MI300X at 10,000 req/s shows \$15.4M/yr in savings. The algorithm adds O(1) dispatch overhead, self-calibrates across content types without a tokenizer, and composes with PagedAttention, continuous batching, and prefill-decode disaggregation.

Related papers

• Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving [0.0]
  We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class.<n>Across three mixed, bursty arrival traces, the router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length.
  arXiv  Detail & Related papers  (2026-09-14T18:39:18Z)
• Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware [1.7286026533711227]
  Masked diffusion language models (dLLMs) can generate text faster than autoregressive (AR) models, since they denoise many tokens at once.<n>Recent systems have begun building serving infrastructure for dLLMs, but none first measure how these models behave under real, concurrent serving.<n>We characterize dLLM serving to close this gap, using LLaDA-8B-Instruct with a D2F (Discrete Diffusion Forcing) LoRA adapter on a single NVIDIA H200 GPU.
  arXiv  Detail & Related papers  (2026-08-24T20:16:48Z)
• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Lower Bounds for PIR with Preprocessing from Blackbox Cryptography [57.20693707878285]
  We study the limits of single-server private information retrieval (PIR) with preprocessing.<n>Our work presents computation lower bounds for any single-server PIR with preprocessing.
  arXiv  Detail & Related papers  (2026-07-07T16:12:48Z)
• SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference [14.525255398490854]
  We propose SparDA, a decoupled sparse attention architecture.<n>Forecast predicts KV blocks needed by the next layer, enabling lookahead selection.<n>SparDA matches or slightly improves accuracy on sparse-pretrained 8B models.
  arXiv  Detail & Related papers  (2026-06-03T06:42:05Z)
• AdaSplash-2: Faster Differentiable Sparse Attention [14.338708749838881]
  We introduce AdaSplash-2, which reduces the number of iterations needed to compute $$ to typically 1--2.<n>AdaSplash-2 matches or improves per-step training time relative to FlashAttention-2 when block sparsity is moderate-to-high.<n>On downstream tasks, models trained with our efficient $$-entmax attention match softmax baselines at short-context lengths and achieve substantial gains in long-context settings.
  arXiv  Detail & Related papers  (2026-04-16T16:03:13Z)
• Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving [9.457255218406333]
  Production vLLM fleets typically provision each instance for the worst-case context length.<n>In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts.<n>We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools.
  arXiv  Detail & Related papers  (2026-04-09T10:47:20Z)
• FlashSampling: Fast and Memory-Efficient Exact Sampling [62.5203057469482]
  FlashSampling is an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in.<n>Across H100, H200, B200, and B300 GPUs, FlashSampling speeds up kernel-level decode workloads.<n>In end-to-end vLLM experiments, it reduces time per output token by up to $19%$ on the models we test.
  arXiv  Detail & Related papers  (2026-03-16T19:37:08Z)
• 98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router [9.457255218406333]
  We present three staged optimizations for the vLLM Semantic Router.<n>A custom Flash Attention operator for ONNX on ROCm reduces attention memory from $O(n2)$ to $O(n)$.<n>Near-streaming body processing with adaptive chunking eliminates serialization overhead.
  arXiv  Detail & Related papers  (2026-03-13T04:33:53Z)
• Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts [68.79341332280062]
  Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time.<n>We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier.<n>Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint.
  arXiv  Detail & Related papers  (2026-02-02T13:52:40Z)
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding [73.67253077506672]
  Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost.<n>Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost.<n>We propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work.
  arXiv  Detail & Related papers  (2025-09-19T04:51:41Z)
• ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models [4.951427498576812]
  We introduce a novel context compression pipeline called Intermediate Layer Retrieval (ILRe)<n>ILRe encodes context by streaming chunked prefill only up to that layer, and recalls tokens by the attention scores between the input query and full key cache in that specified layer.<n>Without additional post training or operator development, ILRe can process a single $1M$ tokens request in less than half a minute.
  arXiv  Detail & Related papers  (2025-08-25T10:59:02Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling [82.08631594071656]
  Pooling layers are essential building blocks of Convolutional Neural Networks (CNNs) We propose an adaptive and exponentially weighted pooling method named adaPool. We demonstrate how adaPool improves the preservation of detail through a range of tasks including image and video classification and object detection.
  arXiv  Detail & Related papers  (2021-11-01T08:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.