KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

• URL: http://arxiv.org/abs/2604.13058v1
• Date: Wed, 18 Mar 2026 01:58:14 GMT
• Title: KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context
• Abstract summary: We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings.<n>KMMMU contains 3,466 questions from exams written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions.<n>Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset.
• Score: 8.518732134730355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings. KMMMU contains 3,466 questions from exams natively written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions. Unlike translated or English-centric benchmarks, KMMMU targets information-dense problems shaped by local conventions, official standards, and discipline-specific visual formats. Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset. Performance varies across disciplines, with some disciplines emerging as bottlenecks, and Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that these failures stem less from insufficient reasoning depth than from weak convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards understanding. KMMMU provides a testbed for multimodal evaluation beyond English-centric benchmarks and for developing more reliable systems for expert real-world tasks.

Related papers

• K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts [43.3460021951349]
  We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems.<n>The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers.<n>On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67%, a substantial drop from BrowseComp.
  arXiv  Detail & Related papers  (2026-06-01T15:50:03Z)
• SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures [93.36078244548077]
  The task data consist of an extended version of our manually constructed BLEnD benchmark (Myung et al. 2024), covering more than 30 language-culture pairs.<n>As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification.<n>We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches.
  arXiv  Detail & Related papers  (2026-05-04T13:49:44Z)
• The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models [51.61416200800499]
  GaoYao is a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas.<n>First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers.<n>Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages.<n>Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs.
  arXiv  Detail & Related papers  (2026-04-22T06:19:46Z)
• CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context [8.678622777553267]
  CAPITU is a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese.<n>The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable.<n>We evaluate 18 state-of-the-art models across single-turn and multi-turn settings.
  arXiv  Detail & Related papers  (2026-03-23T21:16:54Z)
• KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models [36.90941464587649]
  We introduce the Korean Instruction-following Task Evaluation (KITE), a benchmark designed to evaluate both general and Korean-specific instructions.<n>Unlike existing Korean benchmarks that focus mainly on factual knowledge or multiple-choice testing, KITE directly targets diverse, open-ended instruction-following tasks.
  arXiv  Detail & Related papers  (2025-10-17T11:45:15Z)
• SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala [39.525952729268994]
  We introduce SinhalaMMLU, the first multiple-choice question answering benchmark designed specifically for Sinhala.<n>The dataset includes over 7,000 questions spanning secondary to collegiate education levels, aligned with the Sri Lankan national curriculum.<n>We evaluate 26 LLMs on SinhalaMMLU and observe that, while Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies at 67% and 62% respectively, overall model performance remains limited.
  arXiv  Detail & Related papers  (2025-09-03T09:22:39Z)
• MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams [50.293164501645975]
  Multimodal large language models (MLLMs) integrate language and visual cues for problem-solving.<n>Current benchmarks for measuring the intelligence of MLLMs suffer from limited scale, narrow coverage, and unstructured knowledge.<n>We introduce MDK12-Bench, a large-scale multidisciplinary benchmark built from real-world K-12 exams spanning six disciplines.
  arXiv  Detail & Related papers  (2025-08-09T06:21:10Z)
• KoBALT: Korean Benchmark For Advanced Linguistic Tasks [0.6971903955510721]
  KoBALT (Korean Benchmark for Advanced Linguistic Tasks) is a linguistically-motivated benchmark comprising 700 multiple-choice questions.<n>It is designed to advance the evaluation of large language models (LLMs) in Korean.<n>It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora.
  arXiv  Detail & Related papers  (2025-05-22T02:03:07Z)
• R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation [75.33671166231096]
  We introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench)<n>RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing.<n>We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc.
  arXiv  Detail & Related papers  (2025-05-04T07:48:36Z)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models [50.43793764203352]
  We introduce MDK12-Bench, a multi-disciplinary benchmark assessing the reasoning capabilities of MLLMs via real-world K-12 examinations.<n>Our benchmark comprises 140K reasoning instances across diverse difficulty levels from primary school to 12th grade.<n>It features 6,827 instance-level knowledge point annotations based on a well-organized knowledge structure, detailed answer explanations, difficulty labels and cross-year partitions.
  arXiv  Detail & Related papers  (2025-04-08T08:06:53Z)
• MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation [86.7047714187813]
  MMLU-ProX is a benchmark covering 29 languages, built on an English benchmark.<n>Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons.<n>To meet efficient evaluation needs, we provide a lite version containing 658 questions per language.
  arXiv  Detail & Related papers  (2025-03-13T15:59:20Z)
• CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark [53.24896036161829]
  We introduce a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU includes 12k manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. CMMMU focuses on complex perception and reasoning with domain-specific knowledge in the Chinese context.
  arXiv  Detail & Related papers  (2024-01-22T13:34:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.