PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

• URL: http://arxiv.org/abs/2604.11307v1
• Date: Mon, 13 Apr 2026 11:07:08 GMT
• Title: PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers
• Abstract summary: We introduce PaperScope, a benchmark for agentic deep research.<n>It is built on a knowledge graph of over 2,000 AI papers spanning three years.<n>It integrates semantically related key information nodes and employs optimized random-walk article selector.
• Score: 52.97586643334384
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.

Related papers

• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs [68.27437550335709]
  We introduce PaperMind, a benchmark designed to evaluate integrated and agent-oriented scientific reasoning over research papers.<n>PaperMind is constructed from real scientific papers across seven domains, including agriculture, biology, chemistry, computer science, medicine, physics, and economics.<n>By analyzing model behavior across multiple tasks, PaperMind enables a diagnostic evaluation of integrated scientific reasoning behaviors.
  arXiv  Detail & Related papers  (2026-04-23T05:42:39Z)
• Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework [82.66443886385125]
  Paper Circle is a discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature.<n>The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-agent scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes such as concepts, methods, and figures.
  arXiv  Detail & Related papers  (2026-04-07T17:59:58Z)
• EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents [38.845762856105175]
  We introduce EpiBench, an episodic multi-turn benchmark that instantiates short research.<n>Given a research task, agents must navigate across papers over multiple turns, align evidence from figures and tables, and use the accumulated evidence in the memory to answer objective questions.<n>Our experiments show that even the leading model achieves an accuracy of only 29.23% on the hard split, indicating substantial room for improvement.
  arXiv  Detail & Related papers  (2026-04-07T07:58:55Z)
• SciMDR: Benchmarking and Advancing Scientific Multimodal Document Reasoning [59.93568326270048]
  We introduce the synthesize synthesize framework, which generates faithful, isolated QA pairs on focused segments.<n>We construct SciMDR, a large-scale training dataset for crossmodal comprehension.<n>Experiments demonstrate that SciMDR models achieve significant improvements across multiple scientific QA benchmarks.
  arXiv  Detail & Related papers  (2026-03-12T17:57:52Z)
• Doc-Researcher: A Unified System for Multimodal Document Parsing and Deep Research [31.973886754355547]
  Doc-Researcher is a unified system that bridges the gap between text-only, vision-only, and hybrid paradigms.<n>We introduce M4DocBench, the first benchmark for Multi-modal, Multi-hop, Multi-document, and Multi-turn deep research.<n>Experiments demonstrate Doc-Researcher achieves 50.6% accuracy, 3.4xbetter than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-24T16:07:54Z)
• Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines [63.22096609916707]
  Multi-modal Retrieval Augmented Multi-modal Generation (M$2$RAG) is a novel task that enables foundation models to process multi-modal web content.<n>Despite its potential impact, M$2$RAG remains understudied, lacking comprehensive analysis and high-quality data resources.
  arXiv  Detail & Related papers  (2024-11-25T13:20:19Z)
• M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models [27.910693214922052]
  We introduce M3SciQA, a multi-modal, multi-document scientific question answering benchmark for evaluating foundation models. M3SciQA consists of 1,452 expert-annotated questions spanning 70 natural language processing paper clusters. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents.
  arXiv  Detail & Related papers  (2024-11-06T17:52:01Z)
• SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers [43.18330795060871]
  SPIQA is a dataset specifically designed to interpret complex figures and tables within the context of scientific research articles.<n>We employ automatic and manual curation to create the dataset.<n> SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits.
  arXiv  Detail & Related papers  (2024-07-12T16:37:59Z)
• UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding [93.92313947913831]
  We introduce UniDoc, a novel multimodal model equipped with text detection and recognition capabilities. To the best of our knowledge, this is the first large multimodal model capable of simultaneous text detection, recognition, spotting, and understanding.
  arXiv  Detail & Related papers  (2023-08-19T17:32:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.