RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents

• URL: http://arxiv.org/abs/2606.07401v1
• Date: Fri, 05 Jun 2026 15:41:34 GMT
• Title: RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents
• Abstract summary: Document parsing systems are increasingly deployed in high-stakes, regulated domains such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records.<n>Most public benchmarks evaluate adapters on academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score.<n>We introduce RealDocBench, a two-track benchmark built from real regulated documents.
• Score: 0.9003228139607131
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Document parsing systems are increasingly deployed in high-stakes, regulated workflows such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records. Yet most public benchmarks evaluate parsers on clean academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score. Such documents and metrics correlate poorly with what downstream agents actually need: the correct value for a specific field on a messy real-world page. We introduce RealDocBench, a two-track benchmark built from real regulated documents. The QA track contains 1,356 field-level questions over 581 documents spanning four domains, where each question is paired with a typed gold_dict of key-to-value answers and parsers are scored on both per-field and strict per-question accuracy. The layout track contains 1,500 human-verified page images annotated with COCO-style bounding boxes under a nine-class public taxonomy, scored with a Hungarian matcher that includes adjacency-aware split/merge recovery. We evaluate eighteen systems, spanning commercial parsing APIs, general-purpose VLMs, and open-source OCR models, under a uniform extraction-and-scoring protocol, and report accuracy alongside per-page cost and cache-busted latency. RealDocBench exposes a wide performance spread that single-number benchmarks hide, a persistently hard medical sub-domain, and sharp cost/latency trade-offs across operating points. We release the datasets, parser adapters, and evaluation harness to support reproducible, field-level comparison of document parsing systems.

Related papers

• Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing [53.41293908252118]
  We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing.<n>Dr. DocBench spans 52 BISAC domains and selects documents through failure-based sampling.<n>It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality annotations for layout, reading order, hierarchical relations, and domain-specific visual contents.<n>Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.
  arXiv  Detail & Related papers  (2026-05-31T18:35:30Z)
• HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering [19.052998569565627]
  HiKEY is a hierarchical tree-based multimodal retrieval framework that elevates document hierarchy to a first-class retrieval signal.<n> Experiments on ODQA benchmarks demonstrate that HiKEY significantly outperforms page- and chunk-based baselines.
  arXiv  Detail & Related papers  (2026-05-28T08:42:21Z)
• MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing [74.84107522458798]
  MPDocBench-Parse is a benchmark for multi-page document parsing in real-world applications.<n>It contains 433 manually annotated documents with 3,246 pages, covering 15 document types in English and Chinese, with diverse layout styles.
  arXiv  Detail & Related papers  (2026-05-21T07:36:41Z)
• DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding [29.270128057512284]
  We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem.<n>We design a four-stage evaluation protocol that audits each level of the trajectory independently.<n>We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems.
  arXiv  Detail & Related papers  (2026-05-09T11:12:59Z)
• DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM [35.910677096654574]
  Document parsing aims to transform unstructured PDF images into semi-structured data, facilitating the digitization and utilization of information in diverse domains.<n>Common practice often selects the top-performing model on standard benchmarks.<n>We introduce DOCR-Inspector, which formalizes document parsing assessment as fine-grained error detection and analysis.
  arXiv  Detail & Related papers  (2025-12-11T13:16:33Z)
• MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns [80.05126590825121]
  MonkeyOCR v1.5 is a unified vision-language framework that enhances both layout understanding and content recognition.<n>To address complex table structures, we propose a visual consistency-based reinforcement learning scheme.<n>Two specialized modules, Image-Decoupled Table Parsing and Type-Guided Table Merging, are introduced to enable reliable parsing of tables.
  arXiv  Detail & Related papers  (2025-11-13T15:12:17Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links [57.514511353084565]
  We introduce a new domain-agnostic framework for selecting a best-performing approach and annotating cross-document links.<n>We apply our framework in two distinct domains -- peer review and news.<n>The resulting novel datasets lay foundation for numerous cross-document tasks like media framing and peer review.
  arXiv  Detail & Related papers  (2025-09-01T11:32:24Z)
• MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Tasks [56.350173737493215]
  Environmental, Social, and Governance (ESG) reports are essential for evaluating sustainability practices, ensuring regulatory compliance, and promoting financial transparency.<n>MMESGBench is a first-of-its-kind benchmark dataset to evaluate multimodal understanding and complex reasoning across structurally diverse and multi-source ESG documents.<n>MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning across seven distinct document types and three major ESG source categories.
  arXiv  Detail & Related papers  (2025-07-25T03:58:07Z)
• OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations [22.336858733121158]
  We introduce OmniDocBench, a novel benchmark featuring high-quality annotations across nine document sources.<n>We conduct a thorough evaluation of both pipeline-based methods and end-to-end vision-language models.
  arXiv  Detail & Related papers  (2024-12-10T16:05:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.