MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

• URL: http://arxiv.org/abs/2502.12852v1
• Date: Tue, 18 Feb 2025 13:40:05 GMT
• Title: MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
• Abstract summary: We introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates cross-modal and text-only topical matching across 205 languages.<n>Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo.
• Score: 19.13134780850277
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages -- over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.

Related papers

• A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models [50.00033296815133]
  Blind and low-vision (BLV) audiences remain underserved by visual art descriptions.<n>Pilot study investigates curator-guided multilingual art description with Qwen2.5-VL-3B-Instruct for German, Romanian, and Serbian.
  arXiv  Detail & Related papers  (2026-05-29T09:47:45Z)
• Lost in Translation: Do LVLM Judges Generalize Across Languages? [46.119587015038746]
  We introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation.<n> MM-JudgeBench includes over 60K pairwise preference instances spanning 25 typologically diverse languages.<n>By evaluating 22 LVLMs, we uncover substantial cross-lingual performance variance in our proposed benchmark.
  arXiv  Detail & Related papers  (2026-04-21T12:29:10Z)
• DV-VLN: Dual Verification for Reliable LLM-Based Vision-and-Language Navigation [18.493700097379186]
  Vision-and-Language Navigation (VLN) requires an embodied agent to navigate in a complex 3D environment according to natural language instructions.<n>Recent progress in large language models (LLMs) has enabled language-driven navigation with improved interpretability.<n>We propose DV-VLN, a new VLN framework that follows a generate-then-verify paradigm.
  arXiv  Detail & Related papers  (2026-01-26T13:47:55Z)
• Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models [60.39744129890118]
  Large vision-language models (LVLMs) have demonstrated exceptional capabilities in understanding visual information with human languages.<n>In this work, we identify a salient correlation between the multilingual understanding ability of LVLMs and language-specific neuron activations in shallow layers.<n>We introduce PLAST, a training recipe that achieves efficient multilingual enhancement for LVLMs by Precise LAnguage-Specific layers fine-Tuning.
  arXiv  Detail & Related papers  (2025-08-25T18:15:25Z)
• Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation [45.551223552275424]
  Vision-Language Translation is a challenging task that requires accurately recognizing multilingual text embedded in images.<n>We present a comprehensive study of VLT from three key perspectives: data quality, model architecture, and evaluation metrics.
  arXiv  Detail & Related papers  (2025-06-13T14:23:38Z)
• mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks [11.996399504336624]
  We introduce mSTEB, a new benchmark to evaluate the performance of large language models (LLMs) on a wide range of tasks.<n>We evaluate the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B.
  arXiv  Detail & Related papers  (2025-06-10T03:15:08Z)
• MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation [86.7047714187813]
  MMLU-ProX is a benchmark covering 29 languages, built on an English benchmark.<n>Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons.<n>To meet efficient evaluation needs, we provide a lite version containing 658 questions per language.
  arXiv  Detail & Related papers  (2025-03-13T15:59:20Z)
• Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators [38.681443695708786]
  This study provides a comprehensive analysis of the multilingual evaluation performance of 10 recent LLMs. We found that excluding the reference answer from the prompt leads to better performance across various languages. Most LLM-based evaluators show a higher correlation with human judgments in high-resource languages than in low-resource languages.
  arXiv  Detail & Related papers  (2025-03-06T12:04:29Z)
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs [84.24644520272835]
  Large language models (LLMs) showcase varied multilingual capabilities across tasks like translation, code generation, and reasoning. Previous assessments often limited their scope to fundamental natural language processing (NLP) or isolated capability-specific tasks. We present a pipeline for selecting available and reasonable benchmarks from massive ones, addressing the oversight in previous work regarding the utility of these benchmarks. We introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets.
  arXiv  Detail & Related papers  (2024-11-14T01:29:36Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language. Currently, instruction-tuned large language models (LLMs) excel at various English tasks. Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation [62.202893186343935]
  We explore what it would take to adapt Large Language Models for low-resource languages. We show that parallel data is critical during both pre-training andSupervised Fine-Tuning (SFT) Our experiments with three LLMs across two low-resourced language groups reveal consistent trends, underscoring the generalizability of our findings.
  arXiv  Detail & Related papers  (2024-08-23T00:59:38Z)
• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning [12.450293825734313]
  Large language models (LLMs) famously exhibit emergent in-context learning (ICL)<n>This study introduces a benchmark VL-ICL Bench for multimodal in-context learning.<n>We evaluate the abilities of state-of-the-art VLLMs against this benchmark suite.
  arXiv  Detail & Related papers  (2024-03-19T21:31:56Z)
• OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models [59.54423478596468]
  We introduce OMGEval, the first Open-source Multilingual Generative test set that can assess the capability of LLMs in different languages. For each language, OMGEval provides 804 open-ended questions, covering a wide range of important capabilities of LLMs. Specifically, the current version of OMGEval includes 5 languages (i.e., Zh, Ru, Fr, Es, Ar)
  arXiv  Detail & Related papers  (2024-02-21T04:42:41Z)
• Self-Augmented In-Context Learning for Unsupervised Word Translation [23.495503962839337]
  Large language models (LLMs) demonstrate strong word translation or bilingual lexicon induction (BLI) capabilities in few-shot setups. We propose self-augmented in-context learning (SAIL) for unsupervised BLI. Our method shows substantial gains over zero-shot prompting of LLMs on two established BLI benchmarks.
  arXiv  Detail & Related papers  (2024-02-15T15:43:05Z)
• Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models [50.653838482083614]
  This paper introduces a scalable test-bed to assess the capabilities of IT-LVLMs on fundamental computer vision tasks. MERLIM contains over 300K image-question pairs and has a strong focus on detecting cross-modal "hallucination" events in IT-LVLMs.
  arXiv  Detail & Related papers  (2023-12-03T16:39:36Z)
• Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning [27.544311403607786]
  We introduce the Ziya-Visual series, a set of bilingual large-scale vision-language models (LVLMs) Our models adopt the Querying Transformer from BLIP-2, further exploring the assistance of optimization schemes. In addition, we stimulate the understanding ability of GPT-4 in multi-modal scenarios, translating our gathered English image-text datasets into Chinese.
  arXiv  Detail & Related papers  (2023-10-12T09:39:17Z)
• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning [42.68425777473114]
  Vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. We introduce vision-language Model with Multi-Modal In-Context Learning (MMICL), a new approach to allow the VLM to deal with multi-modal inputs efficiently. Our experiments confirm that MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks.
  arXiv  Detail & Related papers  (2023-09-14T17:59:17Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• Meta-learning For Vision-and-language Cross-lingual Transfer [14.594704809280984]
  We propose a novel meta-learning fine-tuning framework for vison-language models. Our framework makes current PVLMs rapidly adaptive to new languages in vision-language scenarios. Our method boosts the performance of current state-of-the-art PVLMs in both zero-shot and few-shot cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-05-24T07:51:42Z)
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis [103.89753784762445]
  Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT) This paper systematically investigates the advantages and challenges of LLMs for MMT. We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4.
  arXiv  Detail & Related papers  (2023-04-10T15:51:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.