Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI

• URL: http://arxiv.org/abs/2604.26382v1
• Date: Wed, 29 Apr 2026 07:48:41 GMT
• Title: Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI
• Abstract summary: We built EnterpriseDocBench to take a swing at parsing fidelity, indexing efficiency, retrieval relevance, and generation groundedness.<n>We ran three pipelines through it -- BM25, dense embedding, and a hybrid -- all with the same GPT-5 generator.<n>One result that genuinely surprised us: factual accuracy on stated claims is 85.5%, but answer completeness averages 0.40.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Most enterprise document AI today is a pipeline. Parse, index, retrieve, generate. Each of those stages has been studied to death on its own -- what's still hard is evaluating the system as a whole. We built EnterpriseDocBench to take a swing at it: parsing fidelity, indexing efficiency, retrieval relevance, and generation groundedness, all on the same corpus. The corpus is built from public, permissively licensed documents across six enterprise domains (five represented in the current pilot). We ran three pipelines through it -- BM25, dense embedding, and a hybrid -- all with the same GPT-5 generator. The headline numbers: hybrid retrieval narrowly beats BM25 (nDCG@5 of 0.92 vs. 0.91), and both beat dense embedding (0.83). Hallucination doesn't grow monotonically with document length -- short documents and very long ones both hallucinate more than medium ones (28.1% and 23.8% vs. 9.2%). Cross-stage correlations are very weak: parsing->retrieval r=0.14, parsing->generation r=0.17, retrieval->generation 0.02. If quality were cascading the way most of us assume, those numbers would be much higher; they aren't. Design caveats are real (parsing fixed, generator shared, automated proxy metrics) and we don't oversell the result. One result that genuinely surprised us: factual accuracy on stated claims is 85.5%, but answer completeness averages 0.40. The system is right when it answers -- it just leaves things out. That gap matters more for real deployments than the headline accuracy number does. We also describe three reference architectures (ColPali, ColQwen2, agentic complexity-based routing) which are not yet integrated end-to-end. Framework, metrics, baselines, and collection scripts will be released open-source on acceptance.

Related papers

• Same Problem, Different Field: Cross-Domain Solution Import via Domain-Stripped Computational Fingerprints [0.0]
  We distill each paper once into a domain- and method-name-stripped faceted computational fingerprint.<n>The goal is solution import: surface cross-field pairs solving the same problem, so a bespoke implementation can be swapped for another field's standard, specialized solver.<n>On a benchmark of 18 method families across 109 papers, the skeleton lifts cross-domain retrieval average precision over the abstract from 0.222 to 0.513, and the whole fingerprint reaches 0.557.
  arXiv  Detail & Related papers  (2026-09-07T15:06:22Z)
• MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities [0.5279475826661643]
  We present Code2 InformationCode Search, a neural retrieval system for semantic code-to-code discovery.<n>By 1.29 million structural (functions, types, and templates) Code across 2,500+ large repositories, our system enables retrieval based on computational intent.<n>We employ a split-build architecture, decouplingintensive offline indexing from a CPU-only serving layer.
  arXiv  Detail & Related papers  (2026-07-29T11:07:07Z)
• BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms [22.217909582731505]
  Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search.<n>We present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold.<n>We measure official accuracy, construction and query tokens, and latency.
  arXiv  Detail & Related papers  (2026-07-29T05:46:11Z)
• When More Documents Hurt RAG: Mitigating Vector Search Dilution with Domain-Scoped, Model-Agnostic Retrieval [2.8753422388037486]
  Retrieval-augmented generation degrades when scaled to large document collections.<n>Top-k retrieval increasingly returns semantically similar but contextually incorrect chunks.<n>We propose MASDR-RAG ( Multi-Agent Scoped Domain Retrieval for RAG) to address this dilution.
  arXiv  Detail & Related papers  (2026-06-09T18:26:24Z)
• Towards Retrieving Interaction Spaces for Agentic Search [62.666902610940525]
  Recent direct corpus (DCI) work shows that agents can interact with the raw corpus through shell tools such as 5.4 interaction and file reads.<n>We argue that the role of retrieval for agentic search is not just to select documents that fit in the LLM context window, but to construct an interaction space.
  arXiv  Detail & Related papers  (2026-06-05T03:47:40Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark [1.9537983097153042]
  Behaviour-Driven Development suites accumulate step-duplicate duplication with documented maintenance cost.<n>We release the largest cross-organisational BDD step corpus to date.
  arXiv  Detail & Related papers  (2026-04-22T11:44:05Z)
• The Amazing Agent Race: Strong Tool Users, Weak Navigators [27.24330141815882]
  Existing tool-use benchmarks for LLM agents are overwhelmingly linear.<n>We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph puzzles (or "legs") with fork-merge tool chains.<n>We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs)
  arXiv  Detail & Related papers  (2026-04-11T15:58:29Z)
• Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework [82.66443886385125]
  Paper Circle is a discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature.<n>The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-agent scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes such as concepts, methods, and figures.
  arXiv  Detail & Related papers  (2026-04-07T17:59:58Z)
• ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge [50.93758649363798]
  Impliret is a benchmark that shifts the reasoning challenge to document-side processing.<n>We evaluate a range of sparse and dense retrievers, all of which struggle in this setting.
  arXiv  Detail & Related papers  (2025-06-17T11:08:29Z)
• MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
  MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
  arXiv  Detail & Related papers  (2025-06-05T16:34:57Z)
• DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
  We propose and name this task emphDocument-Aware Passage Retrieval (DAPR) While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
  arXiv  Detail & Related papers  (2023-05-23T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.