WISERouter: LLM Routing with Workload Budget Constraint
- URL: http://arxiv.org/abs/2607.23765v1
- Date: Sun, 26 Jul 2026 17:20:39 GMT
- Title: WISERouter: LLM Routing with Workload Budget Constraint
- Abstract summary: Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale.<n>We introduce WISE (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration.<n>WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.
- Score: 18.88807929206437
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale. LLM routing exploits diversity in model capability and cost by assigning each query to a suitable model to balance utility and budget. Current methods have two limitations: (i) they either use heuristics that do not always enforce the budget constraint or impose a fixed per-query budget that cannot adapt across the workload and leads to suboptimal performance; (ii) they require supervised learning on a dense dataset with statistics for every query-model pair, which is expensive to collect. To address these challenges, we formulate LLM routing as a constrained contextual multi-armed bandit problem and introduce WISERouter (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration. We further prove that WR-Online achieves a sublinear regret bound of $O(\sqrt{T})$ over a time horizon $T$. Empirical results on RouterBench and SWE-Bench demonstrate that (i) WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and (ii) WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.
Related papers
- CaRL-EM: Cost-Aware Reinforcement Learning for Entity Matching with LLMs [1.3673052804154124]
CaRL-EM learns to dynamically plan the usage of inexpensive and expensive operators based on task complexity.<n>It consistently achieves a better quality-cost trade-off than strong LLM-based baselines.
arXiv Detail & Related papers (2026-09-01T13:05:47Z) - Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints [18.52522897906341]
We study the problem of routing queries to large language models (LLMs)<n>We propose a batch-level, resource-aware routing framework that jointly optimize assignment for each batch.<n> Experiments on two multi-task benchmarks show that improves accuracy by 1-14% on non-robust counterparts.
arXiv Detail & Related papers (2026-03-25T22:24:11Z) - Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
arXiv Detail & Related papers (2026-03-13T12:26:05Z) - MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing [41.77627136743721]
In practical deployments, workloads span lightweight OCR to complex multimodal reasoning.<n>routing is nontrivial due to modality fusion, wide variation in computational cost across models, and the absence of a standardized, budget-aware evaluation.<n>We present MMR-Bench, a unified benchmark that isolates the multimodal routing problem and enables comparison under fixed candidate sets and cost models.
arXiv Detail & Related papers (2026-01-25T12:44:14Z) - xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning [104.63494870852894]
We present x, a tool-calling-based routing system in which a learned router can either answer directly or invoke one or more external models.<n>Our implementation encompasses the full reinforcement learning framework, including reward and cost accounting.<n>Across diverse benchmarks, x achieves strong cost-performance trade-offs.
arXiv Detail & Related papers (2025-10-09T16:52:01Z) - Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs [69.2486294522259]
BaRP is a Bandit Routing-feedback with Preferences approach that trains under the same partial-feedback restriction as deployment.<n> Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt.
arXiv Detail & Related papers (2025-10-08T18:24:59Z) - RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs [51.88834210085435]
We present RADAR (Reasoning-Ability and Difficulty-Aware Routing), a lightweight, interpretable, and scalable routing framework.<n>Inspired by psychometrics, RADAR learns an item response model from model responses with different budgets to different queries.<n>We conduct extensive experiments on 8 widely used reasoning benchmarks, demonstrating the superior performance of RADAR compared to state-of-the-art routing methods.
arXiv Detail & Related papers (2025-09-29T19:33:44Z) - Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving [10.746325451673274]
LLM routing offers a cost-efficient solution by directing queries to the optimal LLM based on model and query features.<n>Existing works primarily focus on offline scenarios and struggle to adapt to online settings.<n>We introduce the first training-free algorithm for online routing scenarios.
arXiv Detail & Related papers (2025-09-02T18:15:03Z) - VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use [78.29315418819074]
We introduce VerlTool, a unified and modular framework that addresses limitations through systematic design principles.<n>Our framework formalizes ARLT as multi-turn trajectories with multi-modal observation tokens (text/image/video), extending beyond single-turn RLVR paradigms.<n>The modular plugin architecture enables rapid tool integration requiring only lightweight Python definitions.
arXiv Detail & Related papers (2025-09-01T01:45:18Z) - Cost-Aware Contrastive Routing for LLMs [57.30288453580456]
We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space.<n>CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%.
arXiv Detail & Related papers (2025-08-17T20:16:44Z) - OmniRouter: Budget and Performance Controllable Multi-LLM Routing [31.60019342381251]
Large language models (LLMs) deliver superior performance but require substantial computational resources and operate with relatively low efficiency.<n>We introduce Omni, a controllable routing framework for multi-LLM serving.<n>Experiments show that Omni achieves up to 6.30% improvement in response accuracy while simultaneously reducing computational costs by at least 10.15%.
arXiv Detail & Related papers (2025-02-27T22:35:31Z) - Cost-Effective Online Multi-LLM Selection with Versatile Reward Models [30.892090566736652]
We introduce the textitC2MAB-V, an online model for selecting and using large language models (LLMs)
textitC2MAB-V is specifically tailored for various collaborative task types with different reward models.
We show that textitC2MAB-V effectively balances performance and cost-efficiency with nine LLMs for three application scenarios.
arXiv Detail & Related papers (2024-05-26T14:38:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.