GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents

• URL: http://arxiv.org/abs/2607.11192v3
• Date: Wed, 15 Jul 2026 07:47:22 GMT
• Title: GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
• Abstract summary: GDP_pdf is a benchmark built to measure document AI capabilities directly.<n>It consists of question-document pairs authored by working professionals in ten fields.<n>We report results for seventeen frontier models on the 100-item benchmark.
• Score: 0.12430801435092285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A large share of day-to-day work in professional domains happens inside PDF files: benefits packets, leases, datasheets, clinical guidelines, construction plans. Benchmarks for document AI have generally measured the required capabilities in isolation: OCR, layout analysis, chart reasoning, table QA, document VQA. A high score on any one of them does not necessarily reveal whether a model can answer a realistic question that someone in the field would actually ask about a specific PDF. GDP_pdf is a benchmark built to measure this directly. It consists of question-document pairs authored by working professionals in ten fields, and a candidate question was kept only when at least two frontier multimodal models failed it in a way that mattered: a wrong answer, missed decisive evidence, or a fabricated claim, rather than a superficial difference such as style. Each item comes with a rubric of atomic criteria, so we can report a graded rubric score as well as a strict task-level pass rate, and each item is tagged against a taxonomy of eleven capabilities in three tiers, spanning text extraction and grounding, table and chart comprehension, cross-referencing, spatial reasoning, and abstention on unsupported queries. We report results for seventeen frontier models on the 100-item benchmark: the best model passes only 30.7% of the items and the worst passes 2%. Most errors trace back to a small set of recurring loss patterns: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text.

Related papers

• XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding [66.6175918865927]
  XL-DocBench is a fully human-verified benchmark for extra-long document understanding.<n>1,519 retained questions from six professional domains and contexts up to 2,303 pages.<n>Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format.
  arXiv  Detail & Related papers  (2026-07-21T09:38:24Z)
• SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows [14.11925465537949]
  We introduce textscSpreadsheetBench 2, a taxonomy-level benchmark for spreadsheet agents.<n>The benchmark is constructed from authentic business data, including financial reports and corporate filings.<n>Results show that current systems remain far from reliable on real-world use.
  arXiv  Detail & Related papers  (2026-06-29T08:33:52Z)
• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents [0.9003228139607131]
  Document parsing systems are increasingly deployed in high-stakes, regulated domains such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records.<n>Most public benchmarks evaluate adapters on academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score.<n>We introduce RealDocBench, a two-track benchmark built from real regulated documents.
  arXiv  Detail & Related papers  (2026-06-05T15:41:34Z)
• Office Comprehension Benchmark [5.899398660180204]
  Office Bench (OCB) is the first public benchmark to jointly evaluate systems on Word, Excel, PowerPoint comprehension over native file formats.<n>OCB consists of two tracks: File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges.<n>Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A.
  arXiv  Detail & Related papers  (2026-05-29T03:19:32Z)
• What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema [0.0]
  We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run.<n>We designed a small audit schema (five fields: benchmark identity, harness specification, inference settings, cost reporting, failure breakdown)<n>We score the disclosure of an agent run, not its correctness, and make no claim that disclosure implies a trustworthy result.
  arXiv  Detail & Related papers  (2026-05-20T17:02:36Z)
• DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding [29.270128057512284]
  We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem.<n>We design a four-stage evaluation protocol that audits each level of the trajectory independently.<n>We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems.
  arXiv  Detail & Related papers  (2026-05-09T11:12:59Z)
• $G^2$-Reader: Dual Evolving Graphs for Multimodal Document QA [53.491241153213565]
  $G2$-Reader is a dual-graph system for multimodal question answering.<n>$G2$-Reader with Qwen3-VL-32B-Instruct reaches 66.21% average accuracy, outperforming strong baselines and a standalone GPT-5 (53.08%)<n>On VisDoMBench across five multimodal domains, $G2$-Reader with Qwen3-VL-32B-Instruct reaches 66.21% average accuracy, outperforming strong baselines and a standalone GPT-5 (53.08%)
  arXiv  Detail & Related papers  (2026-01-29T17:52:54Z)
• Team "better_call_claude": Style Change Detection using a Sequential Sentence Pair Classifier [5.720553544629197]
  At PAN 2025, the shared task challenges participants to detect style at the most fine-grained level: individual sentences.<n>We propose to address this problem by modeling the content of each instance using a Sentence Pair Pair (SSPC) architecture.<n>The model achieves strong macro macro scores of 0.92328, and 0.724 on the EASY MEDIUM, and HARD data, respectively.
  arXiv  Detail & Related papers  (2025-08-01T14:48:17Z)
• MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Tasks [56.350173737493215]
  Environmental, Social, and Governance (ESG) reports are essential for evaluating sustainability practices, ensuring regulatory compliance, and promoting financial transparency.<n>MMESGBench is a first-of-its-kind benchmark dataset to evaluate multimodal understanding and complex reasoning across structurally diverse and multi-source ESG documents.<n>MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning across seven distinct document types and three major ESG source categories.
  arXiv  Detail & Related papers  (2025-07-25T03:58:07Z)
• ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge [50.93758649363798]
  Impliret is a benchmark that shifts the reasoning challenge to document-side processing.<n>We evaluate a range of sparse and dense retrievers, all of which struggle in this setting.
  arXiv  Detail & Related papers  (2025-06-17T11:08:29Z)
• MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations [105.10376440302076]
  This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. It is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models.
  arXiv  Detail & Related papers  (2024-07-01T17:59:26Z)
• PDFTriage: Question Answering over Long, Structured Documents [60.96667912964659]
  Representing structured documents as plain text is incongruous with the user's mental model of these documents with rich structure. We propose PDFTriage that enables models to retrieve the context based on either structure or content. Our benchmark dataset consists of 900+ human-generated questions over 80 structured documents.
  arXiv  Detail & Related papers  (2023-09-16T04:29:05Z)
• Doc2SoarGraph: Discrete Reasoning over Visually-Rich Table-Text Documents via Semantic-Oriented Hierarchical Graphs [79.0426838808629]
  We propose TAT-DQA, i.e. to answer the question over a visually-rich table-text document. Specifically, we propose a novel Doc2SoarGraph framework with enhanced discrete reasoning capability. We conduct extensive experiments on TAT-DQA dataset, and the results show that our proposed framework outperforms the best baseline model by 17.73% and 16.91% in terms of Exact Match (EM) and F1 score respectively on the test set.
  arXiv  Detail & Related papers  (2023-05-03T07:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.