MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing

• URL: http://arxiv.org/abs/2608.00107v1
• Date: Fri, 31 Jul 2026 07:01:21 GMT
• Title: MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing
• Abstract summary: We present MetaRoute-Bench, an open, inspectable framework for comparing meta-decision policies under a shared execution model.<n>A task-aware compositional policy achieves 79.4% success compared with 76.7% for a strong workload-specific static policy.<n>Relative to the static policy, this is a 2.7 percentage-point improvement with paired 95% CI of plus or minus 2.0 points, at 4.7% higher mean cost and 6.4% higher latency.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agentic systems must repeatedly decide whether to answer directly, decompose a task, invoke a tool, execute code, delegate to a specialist, verify an intermediate result, or recover from failure. These meta-decisions affect not only task success but also operating cost and latency, yet they are often embedded inside an orchestration framework and evaluated only through aggregate task accuracy. We present MetaRoute-Bench, an open, inspectable framework for comparing meta-decision policies under a shared execution model. The initial benchmark contains 180 synthetic task profiles spanning data analysis, research, and document processing, eight routing policies, and 30 paired random seeds. Across 43,200 traces, a task-aware compositional policy achieves 79.4% success compared with 76.7% for a strong workload-specific static policy, 67.4% for one-shot task routing, and 52.9% for direct answering. Relative to the static policy, this is a 2.7 percentage-point improvement with paired 95% CI of plus or minus 2.0 points, at 4.7% higher mean cost and 6.4% higher latency. Ablations show the largest losses when route composition is restricted to one operation and when verification is removed. These results are generated by a seeded offline execution model rather than a live deployment; accordingly, the primary contribution is a reproducible evaluation method and an analysis of routing-policy tradeoffs, not evidence of production effectiveness. We release task generation, policies, traces, tests, and analysis artifacts to support live-system validation.

Related papers

• TACIT-Switch: Cost-Aware Model Escalation for LLM Agents from Censored Supervision [2.681967953218225]
  Agents with smaller language-model backbones can drift into persistent failure modes, whereas those with larger backbones are generally more reliable but more costly.<n>This reliability-cost trade-off motivates routing methods that decide when to invoke an agent with a larger backbone.<n>Our method, TACIT-SWITCH, learns permanent handoff policies from accumulated trajectory evidence and Teacher-Annotated Censored Intervention Times.
  arXiv  Detail & Related papers  (2026-08-28T04:33:11Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark [0.0]
  We introduce an executable benchmark and a budget-aware meta-router that composes heterogeneous operations from raw task text.<n>Results: learned success falls to 75.9% and trails static routing at 93.5%, while remaining 49% cheaper and exceeding one-shot routing by 34.3 points.
  arXiv  Detail & Related papers  (2026-07-31T07:00:49Z)
• Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction [62.42945715963878]
  Repository-level vulnerability reproduction is a demanding software engineering (SE) task.<n>Recent LLM agents can often execute these steps when the approach is correct, yet they still fail by choosing the wrong strategy.<n>This paper argues that strategy, rather than the full action trajectory, is the right learning unit for such SE agents.
  arXiv  Detail & Related papers  (2026-07-02T06:27:27Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation [0.0]
  Infrastructure-as-Code (IaC) generation from natural language requires satisfying provider schemas, dependency planning, and organizational policy constraints.<n>We present a verifier-first empirical study of seven agentic strategies for Terraform generation evaluated on IaC-Eval v2, a modernized 186-task AWS/Terraform benchmark with Rego v1 policies.
  arXiv  Detail & Related papers  (2026-05-29T12:43:27Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation [11.272830796781925]
  We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks.<n>Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass.<n>We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:00:57Z)
• Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode [9.116800340266066]
  Claude Code's auto mode is the first deployed permission system for AI coding agents.<n>Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic.<n>We evaluate 253 state-changing actions at the individual action level against oracle ground truth.
  arXiv  Detail & Related papers  (2026-04-04T17:56:30Z)
• Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison [17.732982117200425]
  Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts.<n>This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice.
  arXiv  Detail & Related papers  (2026-03-13T21:47:48Z)
• Semantic Consensus: Process-Aware Conflict Detection and Resolution for Enterprise Multi-Agent LLM Systems [0.0]
  Multi-agent large language model (LLM) systems are rapidly emerging as the dominant architecture for enterprise AI automation.<n>Production deployments exhibit failure rates between 41% and 86.7%, with nearly 79% of failures originating from specification and coordination issues.<n>We propose the Semantic Consensus Framework (SCF), a process-aware comprising six components.
  arXiv  Detail & Related papers  (2026-03-13T14:55:38Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.