How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings

• URL: http://arxiv.org/abs/2605.07492v1
• Date: Fri, 08 May 2026 09:30:31 GMT
• Title: How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings
• Abstract summary: The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
• Score: 56.70440596502351
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The past year has seen over 20 open-source document parsing models, yet thefield still benchmarks almost exclusively on OmniDocBench, a 1,355-pagemanually annotated dataset whose top scores have saturated above 90%. Athree-stage audit pipeline we run on OmniDocBench screens its 21,353evaluator-scored blocks and confirms 2,580 errors (12.08%); combined with overa year of public availability, both annotation quality and contamination riskcall its rankings into question. To address these issues, we presentPureDocBench, a programmatically generated, source-traceable benchmark thatrenders document images from HTML/CSS and produces verifiable annotations fromthe same source, covering 10 domains, 66 subcategories, and 1,475 pages, eachin three versions: clean, digitally degraded, and real-degraded (4,425 imagestotal). Evaluating 40 models spanning pipeline specialists, end-to-endspecialists, and general-purpose VLMs, we find: (i) document parsing is farfrom solved: the best model scores only ~74 out of 100, with a 44.6-point gapbetween the strongest and weakest models; (ii) specialist parsers with <=4Bparameters rival or surpass general VLMs that are 5-100x larger, yet formularecognition remains a shared bottleneck where no model exceeds 67% whenaveraging the formula metric across all three tracks; (iii) general VLMs loseonly 0.99/8.52 Overall points under digital/real degradation versus 4.90/14.21for pipeline specialists, producing ranking reversals that make clean-onlyevaluation misleading for deployment. All data, code, and artifacts arepublicly released.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics [15.233613140316791]
  SemVerBench is the first benchmark of version-constraint resolution semantics across three ecosystems.<n>We evaluate six frontier models and find no statistically significant self-favoritism.<n>Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches 100%.
  arXiv  Detail & Related papers  (2026-09-10T07:34:44Z)
• WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans [5.993812625110576]
  We present a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios.<n>We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence.
  arXiv  Detail & Related papers  (2026-08-24T08:25:12Z)
• From Diagnosis to Correction: Benchmarking and Improving Real-World Table Parsing [50.70862449978062]
  Recent documents achieve table TEDS scores above 93 on OmniDocBench v1.6, yet community feedback and our audit reveal persistent failures on complex real-world tables.<n>To quantify this gap, we introduce TableParseMap, a diagnostic benchmark of 916 real-world tables organized into five challenging scenarios and nine failure types.
  arXiv  Detail & Related papers  (2026-08-10T16:59:30Z)
• WCXB: A Multi-Type Web Content Extraction Benchmark [0.0]
  We introduce the Web Content Extraction Benchmark (WCXB), a dataset of 2,008 web pages from 1,613 domains.<n>The dataset includes a 1,497-page development set and a 511-page held-out test set with matched page type distributions.<n>We evaluate 13 extraction systems and find that while top systems converge on articles, performance diverges sharply on structured page types.
  arXiv  Detail & Related papers  (2026-05-20T12:28:12Z)
• DocAtlas: Multilingual Document Understanding Across 80+ Languages [58.715440331861295]
  We introduce DocAtlas, a framework that constructs high-fidelity OCR datasets and benchmarks covering 82 languages.<n>Our dual pipelines, differential rendering of native DOCX documents and synthetic-based generation for right-to-left scripts produce precise structural annotations.
  arXiv  Detail & Related papers  (2026-05-12T18:09:38Z)
• Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI [0.0]
  We built EnterpriseDocBench to take a swing at parsing fidelity, indexing efficiency, retrieval relevance, and generation groundedness.<n>We ran three pipelines through it -- BM25, dense embedding, and a hybrid -- all with the same GPT-5 generator.<n>One result that genuinely surprised us: factual accuracy on stated claims is 85.5%, but answer completeness averages 0.40.
  arXiv  Detail & Related papers  (2026-04-29T07:48:41Z)
• BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation [34.429649156970015]
  Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
  arXiv  Detail & Related papers  (2026-04-03T16:30:58Z)
• MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios [72.8160644291677]
  We introduce Multilingual Document Parsing Benchmark, the first benchmark for multilingual digital and photographed document parsing.<n> MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions.
  arXiv  Detail & Related papers  (2026-03-30T07:47:46Z)
• VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents [1.06378109904813]
  VAREX is a benchmark for evaluating structured data extraction from government forms.<n>The benchmark comprises 1,777 documents with 1,771 unique documents with validated ground truth through three-phase quality assurance.<n>Results reveal that below 4B parameters, compliance output -- not extraction capability -- is a dominant bottleneck.
  arXiv  Detail & Related papers  (2026-03-16T11:15:56Z)
• AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents [7.014776899553499]
  We present AIForge-Doc, the first dedicated benchmark targeting exclusively diffusion-model-based inpainting in financial and form documents with pixel-level annotation.<n>We benchmark three representative detectors -- TruFor, DocTamper, and a zero-shot GPT-4o judge -- and find that all existing methods degrade substantially.
  arXiv  Detail & Related papers  (2026-02-24T05:37:35Z)
• AI Transparency Atlas: Framework, Scoring, and Real-Time Model Card Evaluation Pipeline [2.1787849426740364]
  We analyzed documentation from five frontier models (Gemini 3, Grok 4.1, Llama 4, GPT-5, and Claude 4.5) and 100 Hugging Face model cards.<n>We developed a weighted transparency framework with 8 sections and 23 subsections that prioritizes safety-critical disclosures.
  arXiv  Detail & Related papers  (2025-12-13T19:48:44Z)
• DocReward: A Document Reward Model for Structuring and Stylizing [107.03974018371058]
  DocReward is a document reward model that evaluates documents based on their structure and style.<n>It is trained using the Bradley-Terry loss to score documents, penalizing predictions that contradict the annotated ranking.<n>It achieves a significantly higher win rate of 60.8%, compared to GPT-5's 37.7% win rate.
  arXiv  Detail & Related papers  (2025-10-13T13:36:32Z)
• MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations [105.10376440302076]
  This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. It is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models.
  arXiv  Detail & Related papers  (2024-07-01T17:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.