The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models

• URL: http://arxiv.org/abs/2604.25359v1
• Date: Tue, 28 Apr 2026 08:27:01 GMT
• Title: The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
• Abstract summary: We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities.<n>All models receive a text-normalized representation of their context regardless of source modality.<n>Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text.
• Score: 0.23332469289621785
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw vision or speech-processing quality, ensuring a fair, source-agnostic comparison. Our benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the dataset, evaluation pipeline, and all related code.

Related papers

• MonkeyOCRv2: A Visual-Text Foundation Model for Document AI [71.06391669976786]
  MonkeyOCRv2 is a visual-text pretrained model for document AI.<n> MonkeyDoc v2 is the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages.
  arXiv  Detail & Related papers  (2026-07-13T13:43:39Z)
• METATR: A Multilingual, Evolving Benchmark for Automatic Text Recognition [1.5803208833562958]
  We introduce METATR (v1.0), a benchmark designed to evaluate Automatic Text Recognition (ATR) models across a wide range of documents.<n>Results are reported across various dimensions, including performance at the dataset and language levels.<n> METATR provides a multidimensional, practitioner-oriented framework for assessing multilingual ATR in real-world conditions.
  arXiv  Detail & Related papers  (2026-05-26T08:53:34Z)
• MinerU-Popo: Universal Post-Processing Model for Structured Document Parsing [34.19535115746437]
  MinerU-Popo is a lightweight framework for POst-Processing OCR outputs.<n>It decomposes the problem into four focused subtasks: text truncation recovery, table truncation recovery, title hierarchy, and image-text association.<n>It improves titlehierarchy TEDS by at least 20% across all five tested OCR models.
  arXiv  Detail & Related papers  (2026-05-24T10:00:28Z)
• VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents [1.06378109904813]
  VAREX is a benchmark for evaluating structured data extraction from government forms.<n>The benchmark comprises 1,777 documents with 1,771 unique documents with validated ground truth through three-phase quality assurance.<n>Results reveal that below 4B parameters, compliance output -- not extraction capability -- is a dominant bottleneck.
  arXiv  Detail & Related papers  (2026-03-16T11:15:56Z)
• Multimodal OCR: Parse Anything from Documents [72.69545534962234]
  dots.mocr treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets.<n>It reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction.<n>It supports end-to-end training over heterogeneous document elements.
  arXiv  Detail & Related papers  (2026-03-13T14:42:21Z)
• UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters [55.34921520578968]
  Vision-language models (VLMs) have achieved impressive unified recognition of text and formulas.<n>We propose UniRec-0.1B, a unified recognition model with only 0.1B parameters.<n>It is capable of performing text and formula recognition at multiple levels, including characters, words, lines, paragraphs, and documents.
  arXiv  Detail & Related papers  (2025-12-24T10:35:21Z)
• Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora [1.7590081165362783]
  We present our pipeline Wasm for processing the Common Crawl dataset to create a new Arabic multimodal dataset.<n>Unlike existing Arabic corpora that focus solely on text extraction, our approach preserves the structural integrity of web content.<n>We provide a comprehensive comparative analysis of our data processing pipeline against those used for major existing datasets.
  arXiv  Detail & Related papers  (2025-11-10T13:10:31Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• Towards Knowledge-Aware Document Systems: Modeling Semantic Coverage Relations via Answerability Detection [40.12543056558646]
  We introduce a novel framework for modelling Semantic Coverage Relations (SCR), which classifies document pairs based on how their informational content aligns.<n>We define three core relation types: equivalence, inclusion, and semantic overlap.<n>We use a question answering (QA)-based approach, using the answerability of shared questions across documents as an indicator of semantic coverage.
  arXiv  Detail & Related papers  (2025-09-10T06:00:01Z)
• OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents [122.55393759474181]
  We introduce OBELICS, an open web-scale filtered dataset of interleaved image-text documents. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. We train vision and language models of 9 and 80 billion parameters named IDEFICS, and obtain competitive performance on different multimodal benchmarks.
  arXiv  Detail & Related papers  (2023-06-21T14:01:01Z)
• DUBLIN -- Document Understanding By Language-Image Network [37.42637168606938]
  We propose DUBLIN, which is pretrained on web pages using three novel objectives. We show that DUBLIN is the first pixel-based model to achieve an EM of 77.75 and F1 of 84.25 on the WebSRC dataset. We also achieve competitive performance on RVL-CDIP document classification.
  arXiv  Detail & Related papers  (2023-05-23T16:34:09Z)
• Rethinking Text Line Recognition Models [57.47147190119394]
  We consider two decoder families (Connectionist Temporal Classification and Transformer) and three encoder modules (Bidirectional LSTMs, Self-Attention, and GRCLs) We compare their accuracy and performance on widely used public datasets of scene and handwritten text. Unlike the more common Transformer-based models, this architecture can handle inputs of arbitrary length.
  arXiv  Detail & Related papers  (2021-04-15T21:43:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.