Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs

• URL: http://arxiv.org/abs/2510.07429v1
• Date: Wed, 08 Oct 2025 18:24:59 GMT
• Title: Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
• Abstract summary: BaRP is a Bandit Routing-feedback with Preferences approach that trains under the same partial-feedback restriction as deployment.<n> Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt.
• Score: 69.2486294522259
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Efficient use of large language models (LLMs) is critical for deployment at scale: without adaptive routing, systems either overpay for strong models or risk poor performance from weaker ones. Selecting the right LLM for each query is fundamentally an online decision problem: models differ in strengths, prices fluctuate, and users value accuracy and cost differently. Yet most routers are trained offline with labels for all candidate models, an assumption that breaks in deployment, where only the outcome of the chosen model is observed. We bridge this gap with BaRP, a Bandit-feedback Routing with Preferences approach that trains under the same partial-feedback restriction as deployment, while supporting preference-tunable inference: operators can dial the performance/cost trade-off at test time without retraining. Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt, rather than depending on full-information offline supervision. Comprehensive experiments show that our method consistently outperforms strong offline routers by at least 12.46% and the largest LLM by at least 2.45%, and generalizes robustly for unseen tasks.

Related papers

• CaRL-EM: Cost-Aware Reinforcement Learning for Entity Matching with LLMs [1.3673052804154124]
  CaRL-EM learns to dynamically plan the usage of inexpensive and expensive operators based on task complexity.<n>It consistently achieves a better quality-cost trade-off than strong LLM-based baselines.
  arXiv  Detail & Related papers  (2026-09-01T13:05:47Z)
• Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning [87.18590662029432]
  We propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models.<n>Our draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query.<n>These ratings enable fine-grained, instance-level textbfProactive Routing and substantially accelerate inference without compromising overall performance.
  arXiv  Detail & Related papers  (2026-06-29T12:30:24Z)
• Learning to Route LLMs from Implicit Cost-Performance Preferences via Meta-Learning [8.795547377156174]
  Large language models (LLMs) present a trade-off between performance and cost, where more powerful models incur greater expense.<n>Existing methods cannot perform well for different user cost-performance preferences.<n>We introduce a novel perceptive LLM routing paradigm for personalized and user-centric cost-performance optimization.
  arXiv  Detail & Related papers  (2026-06-04T13:53:03Z)
• Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection [57.3886742625188]
  Pre-Route is a proactive routing framework that performs structured reasoning before answering.<n>Our study shows three key findings: (i) LLMs possess latent routing ability that can be reliably elicited with guidelines; (ii) linear probes reveal that structured prompts sharpen the separability of the "optimal routing" in representation space; and (iii) distillation transfers this reasoning structure to smaller models for lightweight deployment.
  arXiv  Detail & Related papers  (2026-05-11T09:10:55Z)
• DiSRouter: Distributed Self-Routing for LLM Selections [23.38983740640377]
  We introduce DiS (Distributed Self-), a novel paradigm that shifts from centralized control to distributed routing.<n>In DiS, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness.<n>Extensive experiments demonstrate that DiS significantly outperforms existing routing methods in utility across various scenarios.
  arXiv  Detail & Related papers  (2025-10-22T03:36:40Z)
• xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning [104.63494870852894]
  We present x, a tool-calling-based routing system in which a learned router can either answer directly or invoke one or more external models.<n>Our implementation encompasses the full reinforcement learning framework, including reward and cost accounting.<n>Across diverse benchmarks, x achieves strong cost-performance trade-offs.
  arXiv  Detail & Related papers  (2025-10-09T16:52:01Z)
• LLM Routing with Dueling Feedback [49.67815163970033]
  We study the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost.<n>We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores.<n>We introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting.
  arXiv  Detail & Related papers  (2025-10-01T12:52:25Z)
• Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routing [15.724480880994259]
  A large language model (LLM) router selects the most appropriate model from a pool of candidates for each query.<n> preference-based data, collected via crowdsourcing or LLM-as-a-judge systems, are cheaper and more scalable, yet often biased in reflecting the true quality of responses.<n>We develop an integrative causal router training framework that corrects preference-data bias, address imbalances between two data sources, and improve routing robustness and efficiency.
  arXiv  Detail & Related papers  (2025-09-29T21:44:00Z)
• Cost-Aware Contrastive Routing for LLMs [56.94921736486255]
  We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space.<n>CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%.
  arXiv  Detail & Related papers  (2025-08-17T20:16:44Z)
• How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities [62.474732677086855]
  Large language model (LLM) routing has emerged as a crucial strategy for balancing computational costs with performance.<n>We propose the DSC benchmark: Diverse, Simple, and Categorized, an evaluation framework that categorizes router performance across a broad spectrum of query types.
  arXiv  Detail & Related papers  (2025-03-20T19:52:30Z)
• OmniRouter: Budget and Performance Controllable Multi-LLM Routing [31.60019342381251]
  Large language models (LLMs) deliver superior performance but require substantial computational resources and operate with relatively low efficiency.<n>We introduce Omni, a controllable routing framework for multi-LLM serving.<n>Experiments show that Omni achieves up to 6.30% improvement in response accuracy while simultaneously reducing computational costs by at least 10.15%.
  arXiv  Detail & Related papers  (2025-02-27T22:35:31Z)
• Universal Model Routing for Efficient LLM Inference [69.86195589350264]
  Model routing is a technique for reducing the inference cost of large language models (LLMs)<n>We propose UniRoute, a new approach to the problem of dynamic routing, where new, previously unobserved LLMs are available at test time.<n>We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization [61.02719787737867]
  Large language models (LLMs) are increasingly deployed and democratized on edge devices.<n>One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM.<n>We conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings.
  arXiv  Detail & Related papers  (2025-02-06T18:59:11Z)
• PickLLM: Context-Aware RL-Assisted Large Language Model Routing [0.5325390073522079]
  PickLLM is a lightweight framework that relies on Reinforcement Learning (RL) to route on-the-fly queries to available models.<n>We demonstrate the speed of convergence for different learning rates and improvement in hard metrics such as cost per querying session and overall response latency.
  arXiv  Detail & Related papers  (2024-12-12T06:27:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.