ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

• URL: http://arxiv.org/abs/2607.29677v2
• Date: Wed, 05 Aug 2026 17:48:49 GMT
• Title: ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction
• Abstract summary: We present ExtractBench, a benchmark for schema-guided extraction.<n>ExtractBench scores value accuracy, record completeness at scale, grounding, and measured cost together.<n>The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types.
• Score: 5.372861244667072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.

Related papers

• DocAtlas: Long-Document Understanding as Mutable-State Interaction [67.90400271544678]
  We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process.<n>Given a document and a question, DocAtlas exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence.<n>With GPT-5.4, DocAtlas reaches 71.4% on MMLongBench-Doc, exceeding the human-expert reference of 65.8%.
  arXiv  Detail & Related papers  (2026-07-21T09:45:34Z)
• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents [0.9003228139607131]
  Document parsing systems are increasingly deployed in high-stakes, regulated domains such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records.<n>Most public benchmarks evaluate adapters on academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score.<n>We introduce RealDocBench, a two-track benchmark built from real regulated documents.
  arXiv  Detail & Related papers  (2026-06-05T15:41:34Z)
• Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA [71.42483000929614]
  Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents.<n>We propose Doc-$V*$, an textbfOCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation.
  arXiv  Detail & Related papers  (2026-04-15T11:12:27Z)
• DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion [5.342168661302001]
  We propose a novel framework for controllable synthetic document generation using Vision-Language Models (VLMs)<n>Our approach generates visually plausible and semantically consistent synthetic documents that follow the distribution of an existing source dataset.<n>We show that our framework achieves on average $87%$ of the performance of the full real-world dataset.
  arXiv  Detail & Related papers  (2026-02-25T11:52:13Z)
• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction [11.272127170490059]
  No end-to-end benchmark evaluates PDF-to-JSON extraction under enterprise-scale schema breadth.<n>No principled methodology captures the semantics of nested extraction.<n> ExtractBench is an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction.
  arXiv  Detail & Related papers  (2026-02-12T18:31:37Z)
• ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links [57.514511353084565]
  We introduce a new domain-agnostic framework for selecting a best-performing approach and annotating cross-document links.<n>We apply our framework in two distinct domains -- peer review and news.<n>The resulting novel datasets lay foundation for numerous cross-document tasks like media framing and peer review.
  arXiv  Detail & Related papers  (2025-09-01T11:32:24Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Document Attribution: Examining Citation Relationships using Large Language Models [62.46146670035751]
  We propose a zero-shot approach that frames attribution as a straightforward textual entailment task.<n>We also explore the role of the attention mechanism in enhancing the attribution process.
  arXiv  Detail & Related papers  (2025-05-09T04:40:11Z)
• PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction [28.205723817300576]
  Document pair extraction aims to identify key and value entities as well as their relationships from visually-rich documents. Most existing methods divide it into two separate tasks: semantic entity recognition (SER) and relation extraction (RE) This paper introduces a novel framework, PEneo, which performs document pair extraction in a unified pipeline.
  arXiv  Detail & Related papers  (2024-01-07T12:48:07Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• Multimodal Tree Decoder for Table of Contents Extraction in Document Images [32.46909366312659]
  Table of contents (ToC) extraction aims to extract headings of different levels in documents to better understand the outline of the contents. We first introduce a standard dataset, HierDoc, including image samples from 650 documents of scientific papers with their content labels. We propose a novel end-to-end model by using the multimodal tree decoder (MTD) for ToC as a benchmark for HierDoc.
  arXiv  Detail & Related papers  (2022-12-06T11:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.