An Effective Router for Vision-Language Model Selection

• URL: http://arxiv.org/abs/2606.08970v1
• Date: Mon, 08 Jun 2026 03:10:46 GMT
• Title: An Effective Router for Vision-Language Model Selection
• Abstract summary: ARMS is a router for vision-language models (VLMs) selection.<n>It enhances input signals with VLM profiles, employs a simple but effective architecture to improve representations of queries and VLM capabilities.<n>We show that ARMs can adapt to a broader VLM space and defeat commercial models like GPT-4o that are hundreds of times larger in scale.
• Score: 13.691326653187636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) with varying performance and resource requirements are widely deployed, making it difficult for users to select the most appropriate one among numerous VLM candidates. Existing work reveals the performance paradox phenomenon in language models and focuses on routing methods to solve it. However, developing a router for VLM selection is still a critical yet challenging problem, which primarily faces: 1) lack of specialized data, 2) ineffective feature representation, and 3) rigid model space and costly adaptation. In this paper, we construct a multimodal dataset for VLM selection, containing the outputs of seven mainstream VLMs on 32,626 unique image-text queries. We then propose ARMS, a router for VLM selection. ARMS enhances input signals with VLM profiles, employs a simple but effective architecture to improve representations of queries and VLM capabilities. To improve ARMS' adaptation to new VLMs, we propose two extension training strategies: incremental training and independent training. Experimental results on both in-distribution and out-of-distribution test sets demonstrate the effectiveness of ARMS. In particular, using our training strategy, ARMs (only 800M in size) can adapt to a broader VLM space and defeat commercial models like GPT-4o that are hundreds of times larger in scale. Our code, models, and datasets are available in the anonymous repository.

Related papers

• UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity [82.84279796262818]
  We propose UniFed-VLM, a unified instruction tuning framework for Vision-Language Models (VLMs)<n>It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters.<n>We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance.
  arXiv  Detail & Related papers  (2026-08-16T04:01:04Z)
• GenRecal: Generation after Recalibration from Large to Small Vision-Language Models [63.27511432647797]
  Vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V.<n>Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V.
  arXiv  Detail & Related papers  (2025-06-18T17:59:49Z)
• LaViDa: A Large Diffusion Language Model for Multimodal Understanding [70.99233885354028]
  LaViDa is a family of Vision-Language Models built on Discrete diffusion models.<n>DMs offer parallel decoding for faster inference and bidirectional context for controllable generation.<n>LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks.
  arXiv  Detail & Related papers  (2025-05-22T16:07:12Z)
• VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making [45.02997774119763]
  Vision-language models (VLMs) extend large language models (LLMs) to multi-modal data.<n>Our work approaches these challenges from an offline-to-online reinforcement learning (RL) perspective.
  arXiv  Detail & Related papers  (2025-05-06T04:51:57Z)
• Vision-Language Model Selection and Reuse for Downstream Adaptation [48.6730325059219]
  This paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL)<n>The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent.
  arXiv  Detail & Related papers  (2025-01-30T11:10:46Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning [12.628697648945298]
  Reinforcement learning (RL) requires either manually specifying a reward function, or learning a reward model from a large amount of human feedback. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language.
  arXiv  Detail & Related papers  (2023-10-19T17:17:06Z)
• Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models [77.2078051555533]
  We propose a novel and affordable solution for the effective VL adaption of large language models (LLMs) Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters. MMA is also equipped with a routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions.
  arXiv  Detail & Related papers  (2023-05-24T11:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.