Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance

• URL: http://arxiv.org/abs/2605.01325v1
• Date: Sat, 02 May 2026 08:42:13 GMT
• Title: Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
• Abstract summary: Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders.<n>We show that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models.<n>This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM?
• Score: 65.36257254806647
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders. While recent works have explored various combinations of vision encoders and LLMs, there still lacks a principled understanding of what makes a vision encoder suitable for VLM alignment. In this paper, we systematically investigate this question via comprehensive experiments on a curated collection of 19 pre-trained vision encoders from diverse sources. We first demonstrate that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models. In fact, these metrics show only weak to moderate correlation with VLM performance. This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM? Through comprehensive analysis, we identify that the structural similarity across modalities plays a crucial but previously overlooked role in vision-encoder selection, which we measure using the Gromov-Wasserstein distance as a proxy. From a theoretical perspective, we show that the learnability of cross-modality mapping can be provably associated with the Gromov-Wasserstein distance. Empirical verification on 60+ full VLM training runs shows that our proposed inference-only metric performs significantly better than alternative model selection strategies and exhibits a much stronger correlation with final VLM performance, thereby enabling efficient and effective prediction of VLM performance before full training.

Related papers

• CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning [79.07656918674684]
  CoME-VL is a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder.<n>Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin.
  arXiv  Detail & Related papers  (2026-04-03T17:59:51Z)
• Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders [40.81958598891815]
  Vision Language Model (VLM) development has largely relied on scaling model size.<n>We present Penguin-VL, whose vision encoder is from a text-only LLM.<n>Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining.
  arXiv  Detail & Related papers  (2026-03-06T18:58:04Z)
• Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities [54.94982467313341]
  Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems.<n>We set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking.
  arXiv  Detail & Related papers  (2025-07-10T15:26:41Z)
• EVEv2: Improved Baselines for Encoder-Free Vision-Language Models [72.07868838411474]
  Existing encoder-free vision-language models (VLMs) are narrowing the performance gap with their encoder-based counterparts.<n>We develop efficient strategies for encoder-free VLMs that rival mainstream encoder-based ones.<n>We show that properly and hierarchically associating vision and language within a unified model reduces interference between modalities.
  arXiv  Detail & Related papers  (2025-02-10T18:59:58Z)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities [30.176918208200604]
  Vision-Language Models (VLMs) have emerged as general purpose tools for addressing a variety of complex computer vision problems. These models have been shown to be highly capable, but also lacking some basic visual understanding skills. This paper sets out to understand the limitations of SoTA VLMs on fundamental visual tasks.
  arXiv  Detail & Related papers  (2024-08-13T08:26:32Z)
• Unveiling Encoder-Free Vision-Language Models [62.52803514667452]
  Existing vision-language models (VLMs) mostly rely on vision encoders to extract visual features followed by large language models (LLMs) for visual-language tasks. We bridge the gap between encoder-based and encoder-free models, and present a simple yet effective training recipe towards pure VLMs. We launch EVE, an encoder-free vision-language model that can be trained and forwarded efficiently.
  arXiv  Detail & Related papers  (2024-06-17T17:59:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.