TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing

• URL: http://arxiv.org/abs/2605.18859v1
• Date: Thu, 14 May 2026 08:58:59 GMT
• Title: TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
• Abstract summary: We introduce TwinBench, a step-level routing benchmark with two tracks.<n>The static track provides 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum, and PinchBench.<n>The dynamic track supplies a harness that runs routers on the full 500-case SWE-bench Verified suite.
• Score: 24.63595849324157
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM routing matters most in long-horizon applications such as coding agents, deep research systems, and computer-use agents, where a single user request triggers many model calls. Routing each call to the cheapest sufficient model can cut costs without sacrificing quality, yet existing router benchmarks evaluate routers only on one-shot prompts. They never expose the router-visible prefix at an intermediate agent step, never test whether a cheaper replacement preserves downstream task success, and often rely on online LLM judges at evaluation time. We introduce TwinRouterBench, a step-level routing benchmark with two tracks. The static track provides 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum, and PinchBench, each paired with an execution-verified target tier estimated under a released downgrade-and-cascade protocol; scoring is deterministic arithmetic over tier labels, trajectory membership, and token costs, with no online evaluator-side LLM judge. The dynamic track supplies a harness that runs routers on the full 500-case SWE-bench Verified suite; in this paper we report a 100-case held-out evaluation disjoint from the static SWE supervision split. At each LLM call the router selects a concrete model from a locked pool, and success is measured by official task resolution and realized API spend. The two tracks support fast offline iteration followed by end-to-end validation under live agent execution. Code and data are available at https://github.com/CommonstackAI/TwinRouterBench.

Related papers

• TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI [8.291830023781403]
  We present TRACE, a task-level routing framework that aligns routing with the unit of supervision.<n>By leveraging delayed task feedback, TRACE- learns routing policies that adapt to the workload.<n>It consistently improves the accuracy-matched trade-off, achieving non-latency frontier points.
  arXiv  Detail & Related papers  (2026-07-24T16:29:06Z)
• SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks [11.50404356970464]
  Large language models (LLMs) embedded in multi-turn agentic harnesses are reshaping software engineering (SWE)<n>But routing every task to a frontier model is wasteful when many issues admit cheap fixes.<n>We introduce SWE--a value-based temporal approach that lets a cheap model run for a few exploratory turns and reads the resulting partial trajectory.<n>We show that SWE- greatly improves the cost efficiency of SWE tasks, while maintaining the majority of the performances of the stronger model.
  arXiv  Detail & Related papers  (2026-06-30T01:46:26Z)
• Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation [0.0]
  The benchmark introduced here contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted labels under a fixed schema.<n>The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting.
  arXiv  Detail & Related papers  (2026-06-27T13:59:46Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• R2V Agent: Teaching SLMs When to Ask for Help [13.959667066774124]
  We introduce textbfR2V-Agent, a risk-calibrated SLM-LLM routing framework for interactive agents.<n>R2V combines four components: a distilled small language model (SLM) policy, a stronger teacher LLM, a lightweight process verifier that scores candidate actions at each step, and a calibrated step-level router.
  arXiv  Detail & Related papers  (2026-05-15T20:10:24Z)
• When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models [27.942734943134983]
  Mixture-of-Experts (MoE) language models route each token to a small subset of experts.<n>We compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory.<n>The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning.
  arXiv  Detail & Related papers  (2026-05-08T05:26:09Z)
• Learning Agent Routing From Early Experience [55.88886987958933]
  Boundary is a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct inference or escalate to an agent.<n>We show that Boundary reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference.
  arXiv  Detail & Related papers  (2026-05-08T03:18:40Z)
• RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm [19.239852494559212]
  We introduce RealRoute, a framework that shifts the paradigm from predictive routing to a robust Retrieve-then-Verify mechanism.<n> Experiments show that RealRoute significantly outperforms predictive baselines in the multi-hop Rag reasoning task.
  arXiv  Detail & Related papers  (2026-03-02T17:57:46Z)
• LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing [44.046399484829635]
  Large language model (LLM) routing assigns each query to the most suitable model from an ensemble.<n>We introduce LLMBench, a large-scale benchmark and unified framework for LLM routing.<n>It comprises over 400K instances from 21 datasets and 33 models.
  arXiv  Detail & Related papers  (2026-01-12T05:01:15Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities [62.474732677086855]
  Large language model (LLM) routing has emerged as a crucial strategy for balancing computational costs with performance.<n>We propose the DSC benchmark: Diverse, Simple, and Categorized, an evaluation framework that categorizes router performance across a broad spectrum of query types.
  arXiv  Detail & Related papers  (2025-03-20T19:52:30Z)
• MasRouter: Learning to Route LLMs for Multi-Agent Systems [14.029698552632107]
  Multi-agent systems powered by Large Language Models (LLMs) have been demonstrated to push the boundaries of LLM capabilities.<n>Current routing methods effectively reduce overhead in single-agent scenarios by customizing LLM selection for each query.<n>We first introduce the problem of Multi-Agent Routing System (MASR), which integrates all components of MAS into a unified routing framework.<n>Mas is (1) high-performing, achieving a $1.8%sim8.2%$ improvement over the state-of-the-art method on MBPP; (2) economical, reducing overhead by up to $52.07%$ compared to S
  arXiv  Detail & Related papers  (2025-02-16T14:00:59Z)
• Tracking by Instance Detection: A Meta-Learning Approach [99.66119903655711]
  We propose a principled three-step approach to build a high-performance tracker. We build two trackers, named Retina-MAML and FCOS-MAML, based on two modern detectors RetinaNet and FCOS. Both trackers run in real-time at 40 FPS.
  arXiv  Detail & Related papers  (2020-04-02T05:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.