Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks

• URL: http://arxiv.org/abs/2608.14641v1
• Date: Tue, 28 Jul 2026 21:29:03 GMT
• Title: Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks
• Abstract summary: We evaluate four router implementations across RouterBench, BFCL v4, tau2-bench, and WebArena.<n>We evaluate 290 frozen tasks against a locked matrix of 2,610 candidate outcomes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic systems increasingly delegate model selection to a router, yet open-source routers are usually evaluated with different tasks, candidate pools, and execution protocols, limiting direct comparison. We present a common measurement protocol and hybrid evaluation of four router implementations across RouterBench, BFCL v4, tau2-bench, and WebArena. We evaluate 290 frozen tasks against a locked matrix of 2,610 candidate outcomes. Three routers emit constant or near-constant tier assignments; only vLLM Semantic Router varies materially with prompt content, and it has the highest observed success rate on none of the four benchmarks. Always-Mid matches Aurelio exactly on three benchmarks and within 0.003 on the fourth. For vLLM, task-level superiority tests detect no task-specific advantage over a share-matched content-blind allocation; equivalence is established only on WebArena at the protocol-declared five-percentage-point margin. The results show that, under these configurations and controls, observed gains track selected-tier composition more closely than demonstrated task-specific targeting. Fixed-tier baselines and selected-tier distributions are therefore necessary controls in router evaluation; the findings are scoped to these configurations, candidate pool, and frozen benchmark samples, not to routing paradigms in general.

Related papers

• SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations [88.71160380389277]
  We propose SW routers, a Similarity-Contractive Window Router for multi-turn large language model routing.<n> Experiments on multi-turn dialogue benchmarks demonstrate that SW consistently surpasses strong baselines.<n>Our results highlight that multi-turn large language model routing requires a joint design of context construction and evaluation.
  arXiv  Detail & Related papers  (2026-09-10T11:45:55Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks [26.364477396804194]
  Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost.<n>Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios.<n>We address these limitations with three contributions: (1)VLM-ExecBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains; (2) SCOPE, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration; and (3) CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance
  arXiv  Detail & Related papers  (2026-08-12T14:49:28Z)
• Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation [0.0]
  The benchmark introduced here contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted labels under a fixed schema.<n>The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting.
  arXiv  Detail & Related papers  (2026-06-27T13:59:46Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• Query-aware Routing for Filtered Approximate Nearest Neighbors Search [2.121524887011041]
  Filtered ANN search is a core primitive in modern vector databases and retrieval-augmented generation.<n>We benchmark all major categorical filtered ANN methods across multiple datasets under three predicates.<n>A lightweight ML model predicts each candidate method's recall on the query.<n>We show that our router achieves state-of-the-art recall and QPS balance across all five validation datasets.
  arXiv  Detail & Related papers  (2026-06-18T07:54:12Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection [70.73201284835498]
  We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers.<n>We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models.<n>We propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering.
  arXiv  Detail & Related papers  (2026-01-14T18:43:32Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.