Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

• URL: http://arxiv.org/abs/2606.17188v2
• Date: Wed, 17 Jun 2026 04:29:54 GMT
• Title: Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
• Abstract summary: We introduce PuMVR, a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts.<n>Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%.<n>Cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation.
• Score: 0.27998963147546146
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.

Related papers

• EMBLEM: Enhancing Multi-script Table Detection through Masking [14.822713626646332]
  We introduce MANDALA (Multi-script Annotated Documents for Table Detection), a dataset of 2,323 table-containing pages spanning 18 languages and 15 scripts across diverse domains.<n>We also propose EMBLEM, a masking-based paradigm for Multi-script Table Detection (MTD)<n>EMBLEM generates masked images that conceal script- and font-specific details, enabling models pre-trained on abundant English documents to focus on script-agnostic page layout.
  arXiv  Detail & Related papers  (2026-09-08T06:57:50Z)
• Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR [0.15293427903448023]
  Current Vision-Language Models operate under a flawed assumption that one language corresponds to a single writing system.<n>This overlooks billions of users of multi-script languages like Punjabi, Serbian, Hindi-Urdu, Kurdish, among many others.<n>We introduce PuMVR, the first benchmark designed to quantify script-dependent bias through 375 culturally grounded image-reasoning tasks.
  arXiv  Detail & Related papers  (2026-06-18T15:11:48Z)
• GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts [58.92151016423978]
  We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR across 100+ Unicode scripts.<n>Our benchmark comprises clean and degraded image variants rendered from real multilingual texts.<n>We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts.
  arXiv  Detail & Related papers  (2026-04-14T17:12:41Z)
• MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios [72.8160644291677]
  We introduce Multilingual Document Parsing Benchmark, the first benchmark for multilingual digital and photographed document parsing.<n> MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions.
  arXiv  Detail & Related papers  (2026-03-30T07:47:46Z)
• Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration [70.84108518476744]
  We show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script.<n>We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.
  arXiv  Detail & Related papers  (2026-01-06T10:45:04Z)
• Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective [42.69954782425797]
  Large Vision-Language Models (LVLMs) have shown promising reasoning capabilities on text-rich images from charts, tables, and documents.<n>This raises the need to evaluate LVLM performance on cross-lingual text-rich visual inputs, where the language in the image differs from the language of the instructions.<n>We introduce XT-VQA (Cross-Lingual Text-Rich Visual Question Answering), a benchmark designed to assess how LVLMs handle language inconsistency between image text and questions.
  arXiv  Detail & Related papers  (2024-12-23T18:48:04Z)
• Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts [50.40191599304911]
  We investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. We propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. Our findings show that the effectiveness of transliteration varies by task type and model size.
  arXiv  Detail & Related papers  (2024-07-02T14:51:20Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods typically align vision encoders with Multimodal Large Language Models (MLLMs) via supervised fine-tuning (SFT)<n>We propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level.<n>We introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models [50.40191599304911]
  We propose TransliCo to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script. We show that Furina outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks.
  arXiv  Detail & Related papers  (2024-01-12T15:12:48Z)
• Does Transliteration Help Multilingual Language Modeling? [0.0]
  We empirically measure the effect of transliteration on Multilingual Language Models. We focus on the Indic languages, which have the highest script diversity in the world. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages.
  arXiv  Detail & Related papers  (2022-01-29T05:48:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.