Office Comprehension Benchmark

• URL: http://arxiv.org/abs/2607.01245v1
• Date: Fri, 29 May 2026 03:19:32 GMT
• Title: Office Comprehension Benchmark
• Abstract summary: Office Bench (OCB) is the first public benchmark to jointly evaluate systems on Word, Excel, PowerPoint comprehension over native file formats.<n>OCB consists of two tracks: File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges.<n>Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A.
• Score: 5.899398660180204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.docx, .xlsx, .pptx) and their variants. OCB consists of two tracks. File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges. Domain Q&A tests expert-level reasoning grounded in real-world industry documents across 12 professional domains, with queries requiring multi-step analysis and synthesis across documents. Each reference answer is decomposed into atomic, binary-gradable claims, and an ensemble of LLM judges scores responses against each claim independently. Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A; increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains. We release the dataset, evaluation tooling, judge prompt, and a public leaderboard.

Related papers

• XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding [66.6175918865927]
  XL-DocBench is a fully human-verified benchmark for extra-long document understanding.<n>1,519 retained questions from six professional domains and contexts up to 2,303 pages.<n>Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format.
  arXiv  Detail & Related papers  (2026-07-21T09:38:24Z)
• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents [0.9003228139607131]
  Document parsing systems are increasingly deployed in high-stakes, regulated domains such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records.<n>Most public benchmarks evaluate adapters on academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score.<n>We introduce RealDocBench, a two-track benchmark built from real regulated documents.
  arXiv  Detail & Related papers  (2026-06-05T15:41:34Z)
• Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing [53.41293908252118]
  We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing.<n>Dr. DocBench spans 52 BISAC domains and selects documents through failure-based sampling.<n>It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality annotations for layout, reading order, hierarchical relations, and domain-specific visual contents.<n>Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.
  arXiv  Detail & Related papers  (2026-05-31T18:35:30Z)
• MoDora: Tree-Based Semi-Structured Document Analysis System [62.01015188258797]
  Semi-structured documents integrate diverse interleaved data elements arranged in various and often irregular layouts.<n>MoDora is an LLM-powered system for semi-structured document analysis.<n> Experiments show MoDora outperforms baselines by 5.97%-61.07% in accuracy.
  arXiv  Detail & Related papers  (2026-02-26T14:48:49Z)
• UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters [55.34921520578968]
  Vision-language models (VLMs) have achieved impressive unified recognition of text and formulas.<n>We propose UniRec-0.1B, a unified recognition model with only 0.1B parameters.<n>It is capable of performing text and formula recognition at multiple levels, including characters, words, lines, paragraphs, and documents.
  arXiv  Detail & Related papers  (2025-12-24T10:35:21Z)
• DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections [23.428084176322866]
  We propose DocHop-QA, a large-scale benchmark for multimodal, multi-document, multi-hop question answering.<n> DocHop-QA is domain-agnostic and incorporates diverse information formats, including textual passages, tables, and structural layout cues.<n>We evaluate DocHop-QA through four tasks spanning structured index prediction, generative answering, and multimodal integration.
  arXiv  Detail & Related papers  (2025-08-20T08:17:45Z)
• MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Tasks [56.350173737493215]
  Environmental, Social, and Governance (ESG) reports are essential for evaluating sustainability practices, ensuring regulatory compliance, and promoting financial transparency.<n>MMESGBench is a first-of-its-kind benchmark dataset to evaluate multimodal understanding and complex reasoning across structurally diverse and multi-source ESG documents.<n>MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning across seven distinct document types and three major ESG source categories.
  arXiv  Detail & Related papers  (2025-07-25T03:58:07Z)
• DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems [99.17123445211115]
  We introduce DocBench, a benchmark to evaluate large language model (LLM)-based document reading systems. Our benchmark involves the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions.
  arXiv  Detail & Related papers  (2024-07-15T13:17:42Z)
• UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis [7.952225508086861]
  In academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats. We introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs.
  arXiv  Detail & Related papers  (2024-06-21T14:29:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.