VL-RouterBench: A Benchmark for Vision-Language Model Routing

• URL: http://arxiv.org/abs/2512.23562v1
• Date: Mon, 29 Dec 2025 16:01:19 GMT
• Title: VL-RouterBench: A Benchmark for Vision-Language Model Routing
• Abstract summary: Multi-model routing has evolved from an engineering technique into essential infrastructure.<n>Existing work lacks a systematic, reproducible benchmark for evaluating vision-language models.<n>We present VL-Bench to assess the overall capability of VLM routing systems systematically.
• Score: 38.88730943495439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible benchmark for evaluating vision-language models (VLMs). We present VL-RouterBench to assess the overall capability of VLM routing systems systematically. The benchmark is grounded in raw inference and scoring logs from VLMs and constructs quality and cost matrices over sample-model pairs. In scale, VL-RouterBench covers 14 datasets across 3 task groups, totaling 30,540 samples, and includes 15 open-source models and 2 API models, yielding 519,180 sample-model pairs and a total input-output token volume of 34,494,977. The evaluation protocol jointly measures average accuracy, average cost, and throughput, and builds a ranking score from the harmonic mean of normalized cost and accuracy to enable comparison across router configurations and cost budgets. On this benchmark, we evaluate 10 routing methods and baselines and observe a significant routability gain, while the best current routers still show a clear gap to the ideal Oracle, indicating considerable room for improvement in router architecture through finer visual cues and modeling of textual structure. We will open-source the complete data construction and evaluation toolchain to promote comparability, reproducibility, and practical deployment in multimodal routing research.

Related papers

• SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks [26.364477396804194]
  Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost.<n>Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios.<n>We address these limitations with three contributions: (1)VLM-ExecBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains; (2) SCOPE, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration; and (3) CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance
  arXiv  Detail & Related papers  (2026-08-12T14:49:28Z)
• LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers [31.402829906814365]
  No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment.<n>We present a unified formulation of LLM routing as a sequential decision process characterized by five components.<n>We develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost.
  arXiv  Detail & Related papers  (2026-08-07T06:46:58Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems [46.00150374727385]
  Large language models (LLMs) have achieved success, but cost and privacy constraints necessitate deploying smaller models locally.<n>We propose RouterXBench, a principled evaluation framework with three dimensions: router ability, scenario alignment, and cross-domain robustness.<n>We introduce ProbeDirichlet, a lightweight router that aggregates cross-layer hidden states via learnable Dirichlet with probabilistic training.
  arXiv  Detail & Related papers  (2026-02-12T12:28:27Z)
• RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routing [37.7721677767453]
  We introduce RAG-Bench, the first dataset and benchmark designed for adaptive RAG routing.<n>RAG-Bench revisits retrieval from a query-corpus compatibility perspective and standardizes five representative RAG paradigms for systematic evaluation.<n> Experiments with DeepSeek-V3 and LLaMA-3.1-8B demonstrate that no single RAG paradigm is universally optimal.
  arXiv  Detail & Related papers  (2026-01-30T20:38:11Z)
• LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing [44.046399484829635]
  Large language model (LLM) routing assigns each query to the most suitable model from an ensemble.<n>We introduce LLMBench, a large-scale benchmark and unified framework for LLM routing.<n>It comprises over 400K instances from 21 datasets and 33 models.
  arXiv  Detail & Related papers  (2026-01-12T05:01:15Z)
• Training Report of TeleChat3-MoE [77.94641922160359]
  This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes.<n>We detail systematic methodologies for operator-level and end-to-end numerical verification accuracy, ensuring consistency across hardware platforms.<n>A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations.
  arXiv  Detail & Related papers  (2025-12-30T11:42:14Z)
• ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models [26.059355108708374]
  We propose ECVL-ROUTER, the first scenario-aware routing framework for Vision-Language Models (VLMs)<n>Our approach introduces a new routing strategy and evaluation metrics that dynamically select the appropriate model for each query based on user requirements.<n>Results show that our approach successfully routes over 80% of queries to the small model while incurring less than 10% drop in problem solving probability.
  arXiv  Detail & Related papers  (2025-10-31T07:46:44Z)
• Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers [3.090041654375235]
  We show that a well-tuned k-Nearest Neighbors (kNN) approach outperforms state-of-the-art learned routers across diverse tasks.<n>Our findings reveal that the locality properties of model performance in embedding space enable simple non-parametric methods to achieve strong routing decisions.
  arXiv  Detail & Related papers  (2025-05-19T01:33:41Z)
• LEMUR Neural Network Dataset: Towards Seamless AutoML [35.57280723615144]
  We introduce LEMUR, an open-source dataset and framework that provides a large collection of PyTorch-based neural networks.<n>Each model follows a unified template, with configurations and results stored in a structured database to ensure consistency.<n>LEMUR aims to accelerate AutoML research, enable fair benchmarking, and reduce barriers to large-scale neural network research.
  arXiv  Detail & Related papers  (2025-04-14T09:08:00Z)
• ViLBench: A Suite for Vision-Language Process Reward Modeling [25.565912785217822]
  This paper first benchmarks current vision large language models (VLLMs) as two types of reward models.<n>We introduce ViLBench, a vision-language benchmark designed to require intensive process reward signals.<n>We preliminarily showcase a promising pathway towards bridging the gap between general VLLMs and reward models.
  arXiv  Detail & Related papers  (2025-03-26T06:38:31Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks [60.5257456681402]
  We study the potential for building universal embeddings capable of handling a wide range of downstream tasks.<n>We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split.<n>Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models.
  arXiv  Detail & Related papers  (2024-10-07T16:14:05Z)
• On the Evaluation and Refinement of Vision-Language Instruction Tuning Datasets [71.54954966652286]
  We try to evaluate the Vision-Language Instruction-Tuning (VLIT) datasets. We build a new dataset, REVO-LION, by collecting samples with higher SQ from each dataset. Remarkably, even with only half of the complete data, the model trained on REVO-LION can achieve the performance comparable to simply adding all VLIT datasets up.
  arXiv  Detail & Related papers  (2023-10-10T13:01:38Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.