DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

• URL: http://arxiv.org/abs/2609.02059v1
• Date: Wed, 02 Sep 2026 03:35:12 GMT
• Title: DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
• Abstract summary: We introduce DocHop, a benchmark for integrated chart-context reasoning in document-style images.<n>In DocHop, the document narrative specifies multi-step compositional constraints, while charts provide the corresponding data values.<n>We show a substantial gap to human contextnotators achieve over 90% accuracy, while the best model reaches only 62.83%.
• Score: 31.706151171213346
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have achieved strong performance on structured visual understanding tasks such as chart and document question answering. However, existing benchmarks typically evaluate these domains in isolation, leaving underexplored a key capability: whether models can use textual context to determine how chart evidence should be selected, interpreted, and aggregated. We introduce DocHop, a benchmark for integrated chart--context reasoning in document-style images. In DocHop, the document narrative specifies multi-step compositional constraints, while charts provide the corresponding data values. Questions are grounded on a semantic reference label defined in the narrative, requiring models to resolve target entities from context before aggregating evidence across multiple charts. To enable systematic evaluation, we construct DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, covering 2,074 examples across six task categories. Experiments on a wide range of proprietary and open-source MLLMs show a substantial gap to human performance: annotators achieve over 90% accuracy, while the best model reaches only 62.83%. Reasoning-enhanced models consistently show improved results, but performance degrades as reasoning complexity increases. Overall, DocHop provides a controlled testbed for challenging multi-hop document reasoning.

Related papers

• DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning [14.841209914774778]
  Long Document Visual Question Answering (LongDocVQA) requires Multimodal Large Language Models (MLLMs) to locate, integrate, and reason over heterogeneous document elements distributed across multiple pages.<n>Existing approaches, including end-to-end MLLMs, often lack explicit mechanisms to represent and verify how grounded evidence is progressively composed during reasoning.<n>We propose DocTrace, a hierarchical framework that progressively performs evidence localization, structured document parsing, and evidence graph reasoning.
  arXiv  Detail & Related papers  (2026-08-04T08:04:59Z)
• Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing [53.41293908252118]
  We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing.<n>Dr. DocBench spans 52 BISAC domains and selects documents through failure-based sampling.<n>It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality annotations for layout, reading order, hierarchical relations, and domain-specific visual contents.<n>Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.
  arXiv  Detail & Related papers  (2026-05-31T18:35:30Z)
• Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA [71.42483000929614]
  Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents.<n>We propose Doc-$V*$, an textbfOCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation.
  arXiv  Detail & Related papers  (2026-04-15T11:12:27Z)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)
• MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval [17.50612953979537]
  MoLoRAG is a logic-aware retrieval framework for multi-modal, multi-page document understanding.<n>It combines semantic and logical relevance to deliver more accurate retrieval.<n>Experiments on four DocQA datasets demonstrate average improvements of 9.68% in accuracy.
  arXiv  Detail & Related papers  (2025-09-06T00:59:28Z)
• DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections [23.428084176322866]
  We propose DocHop-QA, a large-scale benchmark for multimodal, multi-document, multi-hop question answering.<n> DocHop-QA is domain-agnostic and incorporates diverse information formats, including textual passages, tables, and structural layout cues.<n>We evaluate DocHop-QA through four tasks spanning structured index prediction, generative answering, and multimodal integration.
  arXiv  Detail & Related papers  (2025-08-20T08:17:45Z)
• MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Tasks [56.350173737493215]
  Environmental, Social, and Governance (ESG) reports are essential for evaluating sustainability practices, ensuring regulatory compliance, and promoting financial transparency.<n>MMESGBench is a first-of-its-kind benchmark dataset to evaluate multimodal understanding and complex reasoning across structurally diverse and multi-source ESG documents.<n>MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning across seven distinct document types and three major ESG source categories.
  arXiv  Detail & Related papers  (2025-07-25T03:58:07Z)
• Doc-CoB: Enhancing Multi-Modal Document Understanding with Visual Chain-of-Boxes Reasoning [12.17399365931]
  Existing one-pass MLLMs process entire document images without considering query relevance.<n>Inspired by the human coarse-to-fine reading pattern, we introduce Doc-CoB, a simple-yet-effective mechanism that integrates human-style visual reasoning into MLLM.<n>Our method allows the model to autonomously select the set of regions most relevant to the query, and then focus attention on them for further understanding.
  arXiv  Detail & Related papers  (2025-05-24T08:53:05Z)
• Doc2SoarGraph: Discrete Reasoning over Visually-Rich Table-Text Documents via Semantic-Oriented Hierarchical Graphs [79.0426838808629]
  We propose TAT-DQA, i.e. to answer the question over a visually-rich table-text document. Specifically, we propose a novel Doc2SoarGraph framework with enhanced discrete reasoning capability. We conduct extensive experiments on TAT-DQA dataset, and the results show that our proposed framework outperforms the best baseline model by 17.73% and 16.91% in terms of Exact Match (EM) and F1 score respectively on the test set.
  arXiv  Detail & Related papers  (2023-05-03T07:30:32Z)
• Document-Level Relation Extraction with Sentences Importance Estimation and Focusing [52.069206266557266]
  Document-level relation extraction (DocRE) aims to determine the relation between two entities from a document of multiple sentences. We propose a Sentence Estimation and Focusing (SIEF) framework for DocRE, where we design a sentence importance score and a sentence focusing loss. Experimental results on two domains show that our SIEF not only improves overall performance, but also makes DocRE models more robust.
  arXiv  Detail & Related papers  (2022-04-27T03:20:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.