MORE: A Multilingual Document Parsing Benchmark and Evaluation

• URL: http://arxiv.org/abs/2607.02956v1
• Date: Fri, 03 Jul 2026 04:59:12 GMT
• Title: MORE: A Multilingual Document Parsing Benchmark and Evaluation
• Abstract summary: MORE is a large-scale benchmark designed for multilingual document parsing evaluation.<n>It covers 149 languages, making it the most linguistically diverse benchmark to date.<n>All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline.
• Score: 13.789730225958877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an evaluation blind spot concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce MORE, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) Unprecedented Scale: It covers 149 languages, making it the most linguistically diverse benchmark to date; (2) Structural Complexity: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) Data Authenticity: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark's effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.

Related papers

• Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study [15.5103844307963]
  Large language models (LLMs) are increasingly used for the automatic evaluation of generated text.<n>This work explores several strategies for developing multilingual LLMs-as-a-judge.
  arXiv  Detail & Related papers  (2026-05-27T16:33:58Z)
• METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition [1.5803208833562958]
  We introduce METATR (v1.0), a benchmark designed to evaluate Automatic Text Recognition (ATR) models across a wide range of documents.<n>Results are reported across various dimensions, including performance at the dataset and language levels.<n> METATR provides a multidimensional, practitioner-oriented framework for assessing multilingual ATR in real-world conditions.
  arXiv  Detail & Related papers  (2026-05-26T08:53:34Z)
• VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding [49.07705729597171]
  VisR-Bench is a benchmark for question-driven multimodal retrieval in long documents.<n>Our benchmark comprises over 35K high-quality QA pairs across 1.2K documents.<n>We evaluate various retrieval models, including text-based methods, multimodal encoders, and MLLMs.
  arXiv  Detail & Related papers  (2025-08-10T21:44:43Z)
• BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models [44.759766566414626]
  We introduce BenchMAX, a multi-way multilingual evaluation benchmark.<n>To maintain high quality, three distinct native-speaking annotators independently annotate each sample.<n>Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages.
  arXiv  Detail & Related papers  (2025-02-11T08:17:19Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages [105.54207724678767]
  Data scarcity is a crucial issue for the development of highly multilingual NLP systems. We propose XTREME-UP, a benchmark defined by its focus on the scarce-data scenario rather than zero-shot. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies.
  arXiv  Detail & Related papers  (2023-05-19T18:00:03Z)
• Multi-lingual Evaluation of Code Generation Models [82.7357812992118]
  We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages. We are able to assess the performance of code generation models in a multi-lingual fashion.
  arXiv  Detail & Related papers  (2022-10-26T17:17:06Z)
• Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan [0.05277024349608833]
  This work focuses on Catalan with the aim of exploring what extent a medium-sized monolingual language model is competitive with state-of-the-art large multilingual models. We build a clean, high-quality textual Catalan corpus (CaText), train a Transformer-based language model for Catalan (BERTa), and devise a thorough evaluation in a diversity of settings. The result is a new benchmark, the Catalan Language Understanding Benchmark (CLUB), which we publish as an open resource.
  arXiv  Detail & Related papers  (2021-07-16T13:52:01Z)
• X-FACT: A New Benchmark Dataset for Multilingual Fact Checking [21.2633064526968]
  We introduce X-FACT: the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers.
  arXiv  Detail & Related papers  (2021-06-17T05:09:54Z)
• XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization [98.61159823343036]
  We present the Word-in-Context dataset (WiC) for assessing the ability to correctly model distinct meanings of a word. We put forward a large multilingual benchmark, XL-WiC, featuring gold standards in 12 new languages. Experimental results show that even when no tagged instances are available for a target language, models trained solely on the English data can attain competitive performance.
  arXiv  Detail & Related papers  (2020-10-13T15:32:00Z)
• XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning [68.57658225995966]
  Cross-lingual Choice of Plausible Alternatives (XCOPA) is a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods falls short compared to translation-based transfer.
  arXiv  Detail & Related papers  (2020-05-01T12:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.