Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR

• URL: http://arxiv.org/abs/2606.20770v1
• Date: Thu, 18 Jun 2026 15:11:48 GMT
• Title: Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR
• Abstract summary: Current Vision-Language Models operate under a flawed assumption that one language corresponds to a single writing system.<n>This overlooks billions of users of multi-script languages like Punjabi, Serbian, Hindi-Urdu, Kurdish, among many others.<n>We introduce PuMVR, the first benchmark designed to quantify script-dependent bias through 375 culturally grounded image-reasoning tasks.
• Score: 0.15293427903448023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current Vision-Language Models (VLMs) are celebrated for their multilingual capabilities, yet they operate under a flawed assumption: that one language corresponds to a single writing system. This overlooks billions of users of multi-script languages like Punjabi, Serbian, Hindi-Urdu, Kurdish, among many others, for whom a model's capability may be fractured by orthographic bias. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), the first benchmark designed to quantify script-dependent bias through 375 culturally grounded image-reasoning tasks across Punjabi's three active scripts (Gurmukhi, Shahmukhi, Roman). Evaluating 10 state-of-the-art VLMs, we expose a substantial Script Gap: models frequently solve visual puzzles in one script while failing identical tasks in another, with accuracy deltas reaching 16% and Script Consistency Rates (SCR) as low as 24.8%. Crucially, visual input boosts absolute performance but does not close this gap, the relative bias persists. Our analysis suggests reasoning patterns show limited cross-script transferability, and Chain-of-Thought pathways diverge based on script alone. We propose SCR as a core metric for script-agnostic evaluation, challenging current multilingual assessment paradigms and providing a framework for equitable AI.

Related papers

• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation [0.27998963147546146]
  We introduce PuMVR, a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts.<n>Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%.<n>Cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation.
  arXiv  Detail & Related papers  (2026-06-15T18:25:23Z)
• GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts [58.92151016423978]
  We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR across 100+ Unicode scripts.<n>Our benchmark comprises clean and degraded image variants rendered from real multilingual texts.<n>We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts.
  arXiv  Detail & Related papers  (2026-04-14T17:12:41Z)
• MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios [72.8160644291677]
  We introduce Multilingual Document Parsing Benchmark, the first benchmark for multilingual digital and photographed document parsing.<n> MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions.
  arXiv  Detail & Related papers  (2026-03-30T07:47:46Z)
• Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration [70.84108518476744]
  We show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script.<n>We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.
  arXiv  Detail & Related papers  (2026-01-06T10:45:04Z)
• GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models [0.0]
  GanitBench is a benchmark consisting of 1527 vision-only questions covering several topics in Mathematics.<n>We evaluate two closed source models for the same, in zero-shot Chain-of-Thought (CoT) and two-shot CoT settings.<n>GPT-4o mini is found to be the more dominant model on the benchmark, with it's highest average accuracy being 38.15%.
  arXiv  Detail & Related papers  (2025-07-31T18:24:05Z)
• A two-stage transliteration approach to improve performance of a multilingual ASR [1.9511556030544333]
  This paper presents an approach to build a language-agnostic end-to-end model trained on a grapheme set. We performed experiments with an end-to-end multilingual speech recognition system for two Indic languages.
  arXiv  Detail & Related papers  (2024-10-09T05:30:33Z)
• TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models [50.40191599304911]
  We propose TransliCo to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script. We show that Furina outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks.
  arXiv  Detail & Related papers  (2024-01-12T15:12:48Z)
• Revisiting the Role of Language Priors in Vision-Language Models [90.0317841097143]
  Vision-language models (VLMs) are applied to a variety of visual understanding tasks in a zero-shot fashion, without any fine-tuning. We study $textitgenerative VLMs$ that are trained for next-word generation given an image. We explore their zero-shot performance on the illustrative task of image-text retrieval across 8 popular vision-language benchmarks.
  arXiv  Detail & Related papers  (2023-06-02T19:19:43Z)
• Curriculum Script Distillation for Multilingual Visual Question Answering [10.721189858694396]
  We introduce a curriculum based on the source and target language translations to finetune the pre-trained models for the downstream task. We show that target languages that share the same script perform better (6%) than other languages and mixed-script code-switched languages perform better than their counterparts.
  arXiv  Detail & Related papers  (2023-01-17T23:55:50Z)
• Does Transliteration Help Multilingual Language Modeling? [0.0]
  We empirically measure the effect of transliteration on Multilingual Language Models. We focus on the Indic languages, which have the highest script diversity in the world. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages.
  arXiv  Detail & Related papers  (2022-01-29T05:48:42Z)
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages [87.5457337866383]
  We introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. We find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks.
  arXiv  Detail & Related papers  (2022-01-27T18:53:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.