SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks

• URL: http://arxiv.org/abs/2608.12127v2
• Date: Wed, 19 Aug 2026 10:27:45 GMT
• Title: SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks
• Abstract summary: Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost.<n>Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios.<n>We address these limitations with three contributions: (1)VLM-ExecBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains; (2) SCOPE, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration; and (3) CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance
• Score: 26.364477396804194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost. Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios, and employs training objectives that dilute multi-positive signals via softmax normalization without incorporating cost. We address these limitations with three contributions: (1)VLM-ExecRouterBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains with 11 candidate models spanning nearly two orders of magnitude in pricing; (2)SCOPE-Router, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration (random/diagnostic/diversity sampling), enabling new models to join routing without retraining; (3)CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space. Empirically, SCOPE-Router achieves the best Rank Score on all three benchmarks, surpassing the runner-up by 1.84 points under OOD settings and by 6.75 points under doubly OOD open-set evaluation. When applied to four diverse routers, CRM+RCCR improves Rank Score by 1.25--6.21 points.

Related papers

• Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation [0.0]
  The benchmark introduced here contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted labels under a fixed schema.<n>The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting.
  arXiv  Detail & Related papers  (2026-06-27T13:59:46Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• Rubric-Guided Process Reward for Stepwise Model Routing [18.58577344135632]
  Stepwise model routing improves the efficiency of Large Reasoning Models (LRMs) by assigning each reasoning step to a suitable model.<n>Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning.<n>We propose RoRo, a rubric-guided process reward framework for stepwise model routing.
  arXiv  Detail & Related papers  (2026-05-28T03:42:24Z)
• Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection [57.3886742625188]
  Pre-Route is a proactive routing framework that performs structured reasoning before answering.<n>Our study shows three key findings: (i) LLMs possess latent routing ability that can be reliably elicited with guidelines; (ii) linear probes reveal that structured prompts sharpen the separability of the "optimal routing" in representation space; and (iii) distillation transfers this reasoning structure to smaller models for lightweight deployment.
  arXiv  Detail & Related papers  (2026-05-11T09:10:55Z)
• Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts [0.07812854697536452]
  We present a large-scale study of multi-tier LLM routing with 206,000 query-model pairs across six benchmarks.<n>We show that a substantial portion of reported unsolvability stems from evaluation artifacts.
  arXiv  Detail & Related papers  (2026-05-08T07:49:24Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• VL-RouterBench: A Benchmark for Vision-Language Model Routing [38.88730943495439]
  Multi-model routing has evolved from an engineering technique into essential infrastructure.<n>Existing work lacks a systematic, reproducible benchmark for evaluating vision-language models.<n>We present VL-Bench to assess the overall capability of VLM routing systems systematically.
  arXiv  Detail & Related papers  (2025-12-29T16:01:19Z)
• Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation [55.47971671635531]
  Large Language Models (LLMs) have shown remarkable performance on general Question Answering (QA)<n>Retrieval-Augmented Generation (RAG) addresses this limitation by enriching LLMs with external knowledge.<n>Existing systems primarily rely on unstructured documents, while largely overlooking relational databases.
  arXiv  Detail & Related papers  (2025-09-30T22:19:44Z)
• One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selection [3.872690949369412]
  Large language models (LLMs) with varying computational costs and performance profiles present a critical challenge for scalable, cost-effective deployment in real-world applications.<n>We introduce a unified routing framework that leverages a single-head cross-attention mechanism to jointly model query and model embeddings.<n>By explicitly capturing fine-grained query-model interactions, our router predicts both response quality and generation cost, achieving up to 6.6% improvement in Average Improvement in Quality (AIQ) and 2.9% in maximum performance over existing routers.
  arXiv  Detail & Related papers  (2025-09-11T18:29:09Z)
• Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving [10.746325451673274]
  LLM routing offers a cost-efficient solution by directing queries to the optimal LLM based on model and query features.<n>Existing works primarily focus on offline scenarios and struggle to adapt to online settings.<n>We introduce the first training-free algorithm for online routing scenarios.
  arXiv  Detail & Related papers  (2025-09-02T18:15:03Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• Learning to Route Queries Across Knowledge Bases for Step-wise Retrieval-Augmented Reasoning [60.84901522792042]
  Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs)<n>We propose R1, a novel MRAG framework that learns to decide when and where to retrieve knowledge based on the evolving reasoning state.<n>R1- can adaptively and effectively leverage diverse KBs, reducing unnecessary retrievals and improving both efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-05-28T08:17:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.