Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation

• URL: http://arxiv.org/abs/2603.18652v1
• Date: Thu, 19 Mar 2026 09:17:21 GMT
• Title: Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation
• Abstract summary: Existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content.<n>We present a benchmarking framework based on synthetically generated PDFs with precise ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity.
• Score: 15.522790536128694
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task. Code and data: https://github.com/phorn1/pdf-parse-bench Metric study and human evaluation: https://github.com/phorn1/table-metric-study

Related papers

• A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction [0.0]
  This study evaluates the reliability of information extraction approaches from KRS documents using three strategies.<n>Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation.
  arXiv  Detail & Related papers  (2026-03-09T02:09:02Z)
• DTBench: A Synthetic Benchmark for Document-to-Table Extraction [19.499877109720945]
  Document-to-table (Doc2Table) extraction derives structured tables from unstructured documents under a target schema.<n>Existing benchmarks neither explicitly distinguish nor comprehensively cover the diverse capabilities required in Doc2Table extraction.<n>We present DTBench, a synthetic benchmark that adopts a proposed two-level taxonomy of Doc2Table capabilities.
  arXiv  Detail & Related papers  (2026-02-14T14:52:36Z)
• Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs [15.522790536128694]
  We introduce a novel benchmarking framework centered on synthetically generated PDFs with precise ground truth.<n>A key methodological contribution is pioneering LLM-as-a-judge for semantic formula assessment.
  arXiv  Detail & Related papers  (2025-12-10T18:01:50Z)
• DAG-Math: Graph-Guided Mathematical Reasoning in LLMs [54.231935013127206]
  Large Language Models (LLMs) demonstrate strong performance on mathematical problems when prompted with Chain-of-Thought (CoT)<n>We propose modeling CoT as a certain rule-based process over directed acyclic graphs (DAGs)<n>We introduce logical closeness, a metric that quantifies how well a model's CoT trajectory adheres to the DAG structure.
  arXiv  Detail & Related papers  (2025-10-19T21:05:17Z)
• From Surface to Semantics: Semantic Structure Parsing for Table-Centric Document Analysis [9.526986293067576]
  DOTABLER is a table-centric semantic document parsing framework.<n>It delivers comprehensive table-anchored semantic analysis and precise extraction of semantically relevant tables.<n> evaluated on nearly 4,000 pages with over 1,000 tables from real-world PDFs.
  arXiv  Detail & Related papers  (2025-08-14T03:29:51Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Can LLMs Generate Tabular Summaries of Science Papers? Rethinking the Evaluation Protocol [83.90769864167301]
  Literature review tables are essential for summarizing and comparing collections of scientific papers.<n>We explore the task of generating tables that best fulfill a user's informational needs given a collection of scientific papers.<n>Our contributions focus on three key challenges encountered in real-world use: (i) User prompts are often under-specified; (ii) Retrieved candidate papers frequently contain irrelevant content; and (iii) Task evaluation should move beyond shallow text similarity techniques.
  arXiv  Detail & Related papers  (2025-04-14T14:52:28Z)
• PoTable: Towards Systematic Thinking via Stage-oriented Plan-then-Execute Reasoning on Tables [13.823024099178172]
  PoTable is a stage-oriented plan-then-execute reasoning approach that achieves systematic thinking on tables.<n>PoTable can produce reliable table reasoning results with highly accurate, steply commented and completely executable programs.
  arXiv  Detail & Related papers  (2024-12-05T15:54:16Z)
• ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models [58.34560740973768]
  We introduce a framework that leverages language models (LMs) to generate literature review tables. A new dataset of 2,228 literature review tables extracted from ArXiv papers synthesize a total of 7,542 research papers. We evaluate LMs' abilities to reconstruct reference tables, finding this task benefits from additional context.
  arXiv  Detail & Related papers  (2024-10-25T18:31:50Z)
• TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios [51.66718740300016]
  TableLLM is a robust large language model (LLM) with 8 billion parameters.<n>TableLLM is purpose-built for proficiently handling data manipulation tasks.<n>We have released the model checkpoint, source code, benchmarks, and a web application for user interaction.
  arXiv  Detail & Related papers  (2024-03-28T11:21:12Z)
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
  We present GraPPa, an effective pre-training approach for table semantic parsing. We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar. To maintain the model's ability to represent real-world data, we also include masked language modeling.
  arXiv  Detail & Related papers  (2020-09-29T08:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.