OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

• URL: http://arxiv.org/abs/2603.08655v1
• Date: Mon, 09 Mar 2026 17:34:53 GMT
• Title: OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
• Abstract summary: We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning.<n>OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning.<n>We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain.
• Score: 34.493139441463995
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.

Related papers

• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding [66.6175918865927]
  XL-DocBench is a fully human-verified benchmark for extra-long document understanding.<n>1,519 retained questions from six professional domains and contexts up to 2,303 pages.<n>Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format.
  arXiv  Detail & Related papers  (2026-07-21T09:38:24Z)
• GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents [0.12430801435092285]
  GDP_pdf is a benchmark built to measure document AI capabilities directly.<n>It consists of question-document pairs authored by working professionals in ten fields.<n>We report results for seventeen frontier models on the 100-item benchmark.
  arXiv  Detail & Related papers  (2026-07-13T07:44:07Z)
• LLM-Guided Planning for Multi-hop Reasoning over Multimodal Nuclear Regulatory Documents [2.3651168422805027]
  We frame this task as planning: an agent observes evidence collected so far, picks next document fragment to inspect, and stops when the evidence is sufficient.<n>The agent operates over a vectorless document tree using browse, read, and search tools, and maintains a dynamic knowledge graph (KG) as state.<n>On a 200-question benchmark over NuScale Final Safety Analysis Report (FSAR) documents, the system reaches 81.5% accuracy with a fidelity of 0.93.
  arXiv  Detail & Related papers  (2026-06-28T13:45:59Z)
• Office Comprehension Benchmark [5.899398660180204]
  Office Bench (OCB) is the first public benchmark to jointly evaluate systems on Word, Excel, PowerPoint comprehension over native file formats.<n>OCB consists of two tracks: File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges.<n>Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A.
  arXiv  Detail & Related papers  (2026-05-29T03:19:32Z)
• Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation [0.0]
  We compare tested models to the contemporaneous frontier on the Epoch AI Capabilities Index.<n>The gap between these answers is widening at +5.53 ECI/year.<n>Proposed remedies include API-access subsidies and editorial enforcement of reporting frameworks.
  arXiv  Detail & Related papers  (2026-05-05T17:58:35Z)
• ResearchGym: Evaluating Language Model Agents on Real-World AI Research [48.46915933681714]
  We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research.<n>To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL.<n>In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap.
  arXiv  Detail & Related papers  (2026-02-16T19:00:03Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization [10.935436958494245]
  Large language models (LLMs) now support contexts of up to 1M tokens, but their effectiveness on complex long-context tasks remains unclear.<n>We study multi-document legal case summarization, where a single case often spans many documents totaling 100K-500K tokens.<n>We introduce Gavel-Ref, a reference-based evaluation framework with multi-value checklist evaluation over 26 items, as well as residual fact and writing-style evaluations.
  arXiv  Detail & Related papers  (2026-01-07T22:08:17Z)
• DocReward: A Document Reward Model for Structuring and Stylizing [107.03974018371058]
  DocReward is a document reward model that evaluates documents based on their structure and style.<n>It is trained using the Bradley-Terry loss to score documents, penalizing predictions that contradict the annotated ranking.<n>It achieves a significantly higher win rate of 60.8%, compared to GPT-5's 37.7% win rate.
  arXiv  Detail & Related papers  (2025-10-13T13:36:32Z)
• QUEST: Quality-aware Semi-supervised Table Extraction for Business Documents [0.1516287840715525]
  QUEST is a Quality-aware Semi-supervised Table extraction framework for business documents.<n>QUEST evaluates structural and contextual features of extracted tables, trained to predict F1 scores instead of relying on confidence metrics.
  arXiv  Detail & Related papers  (2025-06-17T14:25:44Z)
• SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation [2.985620880452744]
  SciSage is a multi-agent framework employing a reflect-when-you-write paradigm.<n>It critically evaluates drafts at outline, section, and document levels, collaborating with specialized agents for query interpretation, content retrieval, and refinement.<n>We also release SurveyScope, a benchmark of 46 high-impact papers ( 2020-2025) across 11 computer science domains.
  arXiv  Detail & Related papers  (2025-06-15T02:23:47Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
  We propose and name this task emphDocument-Aware Passage Retrieval (DAPR) While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
  arXiv  Detail & Related papers  (2023-05-23T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.