The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers

• URL: http://arxiv.org/abs/2606.07587v1
• Date: Wed, 27 May 2026 19:29:01 GMT
• Title: The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
• Abstract summary: LLM routing is a popular approach to improve the cost-quality trade-off of LLM services by dynamically selecting a model for each query.<n>Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches.<n>Our study reveals a consistent phenomenon that we call the routing plateau: many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router.
• Score: 19.847272668359707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM routing has become a popular approach to improve the cost-quality trade-off of LLM services by dynamically selecting a model for each query. Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches. Our extensive study of 21 routing methods across five benchmarks reveals a consistent phenomenon that we call the routing plateau: many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router. Our investigation shows that the plateau is largely caused by a predictability bottleneck: current routers mainly learn global averaged model-performance trends rather than fine-grained query-specific routing signals. As a result, they solve overlapping easy queries but collectively fail on hard queries that require instance-specific routing decisions. We further study how to move beyond the plateau and find that larger training datasets, stronger encoders, and end-to-end fine-tuning can further improve routing accuracy. These findings characterize the common limits of current routing methods and provide insights and actionable directions for the community to build more effective routing systems.

Related papers

• LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers [31.402829906814365]
  No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment.<n>We present a unified formulation of LLM routing as a sequential decision process characterized by five components.<n>We develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost.
  arXiv  Detail & Related papers  (2026-08-07T06:46:58Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing [41.435534676485936]
  RouteJudge is an online pairwise preference evaluation framework for LLM routing systems.<n>Multiple routing strategies independently recommend model models under the same model pool and budget constraints.<n>The resulting user preferences are attributed back to the routing strategies behind the compared responses.
  arXiv  Detail & Related papers  (2026-06-17T07:35:10Z)
• Learning Agent Routing From Early Experience [55.88886987958933]
  Boundary is a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct inference or escalate to an agent.<n>We show that Boundary reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference.
  arXiv  Detail & Related papers  (2026-05-08T03:18:40Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing [44.046399484829635]
  Large language model (LLM) routing assigns each query to the most suitable model from an ensemble.<n>We introduce LLMBench, a large-scale benchmark and unified framework for LLM routing.<n>It comprises over 400K instances from 21 datasets and 33 models.
  arXiv  Detail & Related papers  (2026-01-12T05:01:15Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering [51.07491603393163]
  tAgent is a framework that formulates multi-agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals.<n>By leveraging soft supervision and weighted aggregation of agent outputs, Agent learns principled collaboration schemes that capture the complementary strengths of diverse agents.
  arXiv  Detail & Related papers  (2025-10-06T23:20:49Z)
• RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models [45.58601993849455]
  Retrieval-Augmented Generation (RAG) significantly improves the performance of Large Language Models (LLMs) on knowledge-intensive tasks.<n>We observe that external documents dynamically affect LLMs' ability to answer queries, while existing routing methods exhibit suboptimal performance in RAG scenarios.<n>We propose RAG, a parametric RAG-aware routing design, which leverages document embeddings and RAG capability embeddings with contrastive learning to capture knowledge representation shifts.
  arXiv  Detail & Related papers  (2025-05-29T03:44:56Z)
• Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers [3.090041654375235]
  We show that a well-tuned k-Nearest Neighbors (kNN) approach outperforms state-of-the-art learned routers across diverse tasks.<n>Our findings reveal that the locality properties of model performance in embedding space enable simple non-parametric methods to achieve strong routing decisions.
  arXiv  Detail & Related papers  (2025-05-19T01:33:41Z)
• Universal Model Routing for Efficient LLM Inference [69.86195589350264]
  Model routing is a technique for reducing the inference cost of large language models (LLMs)<n>We propose UniRoute, a new approach to the problem of dynamic routing, where new, previously unobserved LLMs are available at test time.<n>We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.