ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving

• URL: http://arxiv.org/abs/2604.00136v1
• Date: Tue, 31 Mar 2026 18:41:53 GMT
• Title: ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving
• Abstract summary: LLM serving often relies on multi-model portfolios spanning a 530x cost range.<n>This trade-off is non-stationary; providers revise pricing, model quality can regress silently, and new models must be integrated without downtime.<n>We present an open-source adaptive router built on cost-aware contextual bandits.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Production LLM serving often relies on multi-model portfolios spanning a ~530x cost range, where routing decisions trade off quality against cost. This trade-off is non-stationary: providers revise pricing, model quality can regress silently, and new models must be integrated without downtime. We present ParetoBandit, an open-source adaptive router built on cost-aware contextual bandits that is the first to simultaneously enforce dollar-denominated budgets, adapt online to such shifts, and onboard new models at runtime. ParetoBandit closes these gaps through three mechanisms. An online primal-dual budget pacer enforces a per-request cost ceiling over an open-ended stream, replacing offline penalty tuning with closed-loop control. Geometric forgetting on sufficient statistics enables rapid adaptation to price and quality shifts while bootstrapping from offline priors. A hot-swap registry lets operators add or remove models at runtime, with a brief forced-exploration phase for each newcomer, after which UCB selection discovers its quality-cost niche from live traffic alone. We evaluate ParetoBandit across four deployment scenarios on 1,824 prompts routed through a three-model portfolio. Across seven budget ceilings, mean per-request cost never exceeds the target by more than 0.4%. When conditions shift, the system adapts: an order-of-magnitude price cut on the costliest model yields up to +0.071 quality lift, and a silent quality regression is detected and rerouted within budget. A cold-started model reaches meaningful adoption within ~142 steps without breaching the cost ceiling. The router discriminates rather than blindly adopting: expensive models are budget-gated and low-quality models rejected after bounded exploration. End-to-end routing latency is 9.8ms on CPU -- less than 0.4% of typical inference time -- with the routing decision itself taking just 22.5us.

Related papers

• Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances [5.053749663617978]
  Quantized Mixture-of-Experts (MoE) services can hold several pre-materialized instances of one base model.<n>Because instance materialization and replica counts consume memory, we treat them as upstream provisioning decisions.<n>We route each request to maximize modeled throughput under a class-level expected quality-degradation budget.
  arXiv  Detail & Related papers  (2026-09-11T07:57:25Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks [26.364477396804194]
  Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost.<n>Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios.<n>We address these limitations with three contributions: (1)VLM-ExecBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains; (2) SCOPE, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration; and (3) CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance
  arXiv  Detail & Related papers  (2026-08-12T14:49:28Z)
• HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference [47.0879800334801]
  HetRoute is a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference.<n>HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%.
  arXiv  Detail & Related papers  (2026-08-01T10:30:01Z)
• SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learning [2.6853734738584047]
  SeqRoute is a framework that formulates multi-turn routing as a finite-horizon Markov Decision Process.<n>It learns delayed gratification to strategically preserve resources for high-stakes turns later in the session.<n>It reduces operational costs by 6.0-73.5% while maintaining or improving quality, and suppresses bankruptcy rates to under 1%.
  arXiv  Detail & Related papers  (2026-05-25T04:52:10Z)
• CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference [52.849509991178884]
  Large language models (LLMs) move from centralized clouds to mobile edge environments.<n> query-level routing between lightweight on-device models and stronger edge models provides a flexible mechanism to navigate this trade-off.<n>Existing routers are designed for centralized cloud settings and optimize token-level costs, failing to capture the dynamic latency and energy overheads in wireless edge deployments.
  arXiv  Detail & Related papers  (2026-05-12T11:50:15Z)
• RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization [13.891522069967507]
  RouteNLP is a closed-loop framework that routes queries across a tiered model portfolio to minimize cost while satisfying per-task quality constraints.<n>In an 8-week pilot deployment processing 5K queries/day at an enterprise customer-service division, RouteNLP reduced inference costs by 58% while maintaining 91% response acceptance and reducing p99 latency from 1,847 ms to 387 ms.
  arXiv  Detail & Related papers  (2026-04-26T07:34:36Z)
• RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment [57.588738943463646]
  Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT)<n> deploying them at scale remains prohibitively expensive.<n>We propose textbfRouteLMT, an in-model router that predicts the expected gain by probing the small translators prompt-token representation.
  arXiv  Detail & Related papers  (2026-04-24T13:02:45Z)
• RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents [91.0187958746262]
  RouteMoA is an efficient mixture-of-agents framework with dynamic routing.<n>It employs a lightweight scorer to perform initial screening by predicting coarse-grained performance from the query.<n>It refines these scores through lightweight self- and cross-assessment based on existing model outputs, providing posterior correction without additional inference.
  arXiv  Detail & Related papers  (2026-01-26T04:22:22Z)
• e1: Learning Adaptive Control of Reasoning Effort [88.51897900019485]
  Increasing the thinking budget of AI models can significantly improve accuracy, but not all questions warrant the same amount of reasoning.<n>Users may prefer to allocate different amounts of reasoning effort depending on how they value output quality versus latency and cost.<n>We propose Adaptive Effort Control, a self-adaptive reinforcement learning method that trains models to use a user-specified fraction of tokens.
  arXiv  Detail & Related papers  (2025-10-30T23:12:21Z)
• Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs [69.2486294522259]
  BaRP is a Bandit Routing-feedback with Preferences approach that trains under the same partial-feedback restriction as deployment.<n> Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt.
  arXiv  Detail & Related papers  (2025-10-08T18:24:59Z)
• IPR: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs [19.658944117970137]
  textbfIngent textbfPrompt textbfRouting framework dynamically selects optimal models based on predicted response quality and user-specified tolerance levels.<n>IPR achieves 43.9% cost reduction while maintaining quality parity with the strongest model in the Claude family.
  arXiv  Detail & Related papers  (2025-09-08T01:46:27Z)
• Cost-Aware Contrastive Routing for LLMs [57.30288453580456]
  We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space.<n>CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%.
  arXiv  Detail & Related papers  (2025-08-17T20:16:44Z)
• Keeping Up with the Models: Online Deployment and Routing of LLMs at Scale [6.911384287238722]
  We present a hierarchical algorithm that selects up to $M_max$ models for the next stage using reward upper-confidence and cost lower-confidence bounds.<n>We prove that StageRoute achieves a regret of order $T2/3$ and provide a matching lower bound, thereby establishing its near-optimality.
  arXiv  Detail & Related papers  (2025-06-08T12:25:26Z)
• On Optimal Caching and Model Multiplexing for Large Model Inference [66.50550915522551]
  Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. We study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing.
  arXiv  Detail & Related papers  (2023-06-03T05:01:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.