EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

• URL: http://arxiv.org/abs/2603.27223v1
• Date: Sat, 28 Mar 2026 10:29:51 GMT
• Title: EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams
• Abstract summary: EuraGovExam is a benchmark sourced from real-world civil service examinations across five representative Eurasian regions.<n>The dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains.<n>It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation.
• Score: 0.8399688944263844
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. Designed to reflect the authentic complexity of public-sector assessments, the dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains. Unlike existing benchmarks, EuraGovExam embeds all question content--including problem statements, answer choices, and visual elements--within a single image, providing only a minimal standardized instruction for answer formatting. This design demands that models perform layout-aware, cross-lingual reasoning directly from visual input. All items are drawn from real exam documents, preserving rich visual structures such as tables, multilingual typography, and form-like layouts. Evaluation results show that even state-of-the-art vision-language models (VLMs) achieve only 86% accuracy, underscoring the benchmark's difficulty and its power to diagnose the limitations of current models. By emphasizing cultural realism, visual complexity, and linguistic diversity, EuraGovExam establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation.

Related papers

• SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures [93.36078244548077]
  The task data consist of an extended version of our manually constructed BLEnD benchmark (Myung et al. 2024), covering more than 30 language-culture pairs.<n>As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification.<n>We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches.
  arXiv  Detail & Related papers  (2026-05-04T13:49:44Z)
• SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia [40.4434142867308]
  We introduce SEA-Vision, a benchmark that jointly evaluates Document Parsing and Text-Centric Visual Question Answering (TEC-VQA) across 11 Southeast Asian languages.<n>SEA-Vision contains 15,234 document parsing pages from nine representative document types, annotated with hierarchical page-, block-, and line-level labels.<n>It also provides 7,496 TEC-VQA question-answer pairs that probe text recognition, numerical calculation, comparative analysis, logical reasoning, and spatial understanding.
  arXiv  Detail & Related papers  (2026-03-16T15:21:12Z)
• Multimodal Evaluation of Russian-language Architectures [88.00147763684451]
  We introduce Mera Multi, an open multimodal evaluation framework for Russian-spoken architectures.<n>The benchmark is instruction-based and encompasses default text, image, audio, and video modalities.<n>Mera Multi provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages.
  arXiv  Detail & Related papers  (2025-11-19T15:43:53Z)
• PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models [13.316431293058763]
  We introduce PISA-Bench, a benchmark derived from English examples of the expert-created PISA tests.<n>Each example consists of human-extracted instructions, questions, answer options, and images, enriched with question type categories.<n>We evaluate state-of-the-art vision-language models on PISA-Bench and find that especially small models (20B parameters) fail to achieve high test scores.
  arXiv  Detail & Related papers  (2025-10-27T11:00:45Z)
• BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models [54.16874020794336]
  We introduce BLEnD-Vis, a benchmark designed to evaluate the robustness of everyday cultural knowledge in vision-language models (VLMs)<n> BLEnD-Vis constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats.<n>The resulting benchmark comprises 4,916 images and over 21,000 multiple-choice question (MCQ) instances, validated through human annotation.
  arXiv  Detail & Related papers  (2025-10-13T09:10:05Z)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark [68.21939124278065]
  Culturally-diverse multilingual Visual Question Answering benchmark designed to cover a rich set of languages and cultures. CVQA includes culturally-driven images and questions from across 30 countries on four continents, covering 31 languages with 13 scripts, providing a total of 10k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-06-10T01:59:00Z)
• EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models [29.31649801849329]
  EXAMS-V is a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies. The questions come in 11 languages from 7 language families.
  arXiv  Detail & Related papers  (2024-03-15T15:08:39Z)
• Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams [14.801853435122908]
  We present a framework to evaluate language models on entrance exams, which incorporates both textual and visual elements. We evaluate the two most recent editions of Exame Nacional do Ensino M'edio (ENEM), the main standardized entrance examination adopted by Brazilian universities. One of the highlights is that text captions transcribing visual content outperform the direct use of images, suggesting that the vision model has room for improvement.
  arXiv  Detail & Related papers  (2023-11-23T19:20:59Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages [87.5457337866383]
  We introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. We find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks.
  arXiv  Detail & Related papers  (2022-01-27T18:53:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.