VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval
- URL: http://arxiv.org/abs/2608.17889v1
- Date: Tue, 18 Aug 2026 15:23:06 GMT
- Title: VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval
- Abstract summary: VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
- Score: 65.89247522243959
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Visually rich documents encode relevance through language, layout, structured visual elements, and corpus context, yet retrieval is typically evaluated by one-shot query--page matching. Agentic-search benchmarks usually score downstream question answering or report generation, leaving document ranking under iterative evidence acquisition underexplored. We introduce VisDocAgentBench, a closed-corpus benchmark comparing static and agentic retrieval under a shared ranked-output contract. It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures. Relation-preserving construction yields semantic, relational, and visual queries, followed by full-document review and hard-negative validation. A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context. Agents recover much of this loss, but planner choice and retrieval representation remain decisive. Every planner performs better with visual retrieval, whose best R@1 reaches 67.50% versus 37.50% for OCR-text. Ablations identify iterative search and page inspection as consequential capabilities, and providing the complete support context improves ranking on both routes. Trace analysis localizes the remaining losses to target discovery, candidate examination, and evidence-role integration. These findings motivate retrieval agents that combine modality-preserving discovery with evidence-directed verification.
Related papers
- Group-Aware Adaptive Retrieval for Evidence Navigation [10.083876575737222]
Reasoning-intensive retrieval addresses queries whose relevance cannot be identified by surface-level matching.<n>Existing methods iteratively expand a candidate pool at the document level over a corpus graph.<n>We propose Group-Aware Adaptive Retrieval for Evidence Navigation (GAREN) to explore the corpus graph through group-level expansion.
arXiv Detail & Related papers (2026-09-02T06:51:06Z) - DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments [51.049999642138]
We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence.<n>Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly.<n>Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops.
arXiv Detail & Related papers (2026-07-19T15:20:40Z) - HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering [19.052998569565627]
HiKEY is a hierarchical tree-based multimodal retrieval framework that elevates document hierarchy to a first-class retrieval signal.<n> Experiments on ODQA benchmarks demonstrate that HiKEY significantly outperforms page- and chunk-based baselines.
arXiv Detail & Related papers (2026-05-28T08:42:21Z) - Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
arXiv Detail & Related papers (2026-03-09T18:20:20Z) - Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval [68.71038700559195]
Chain of Retrieval(COR) is a novel iterative framework for full-paper retrieval.<n>We present SCIBENCH, a benchmark providing both complete and segmented contexts of full papers for queries and candidates.
arXiv Detail & Related papers (2025-07-14T08:41:53Z) - BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
arXiv Detail & Related papers (2024-07-16T17:58:27Z) - DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
We propose and name this task emphDocument-Aware Passage Retrieval (DAPR)
While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context.
Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
arXiv Detail & Related papers (2023-05-23T10:39:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.