SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding

• URL: http://arxiv.org/abs/2609.05141v1
• Date: Fri, 04 Sep 2026 13:39:48 GMT
• Title: SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
• Abstract summary: We introduce SciDocBench, a workflow-centered benchmark for scientific document understanding.<n>It contains 124 expert-authored and difficulty-screened questions organized into seven research-assistant capability groups and 19 subtasks across five scientific domains.<n>The strongest evaluated system achieves only 62.6/100, with pronounced weaknesses in document perception, evidence grounding, verification, and cross-document reasoning.<n>Building on SciDocIR, we construct SciDocDataset, comprising approximately 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks.
• Score: 51.537009919690696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific papers require models to reason jointly over text, equations, figures, tables, code, and datasets while preserving the provenance of supporting evidence. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear whether multimodal models can support realistic scientific-reading workflows. We introduce SciDocBench, a workflow-centered benchmark for scientific document understanding. It contains 124 expert-authored and difficulty-screened questions organized into seven research-assistant capability groups and 19 subtasks across five scientific domains. Each question is instantiated under four matched conditions combining English or Chinese questions with all-images-first or interleaved document representations, yielding 496 evaluation instances for controlled analysis. The strongest evaluated system achieves only 62.6/100, with pronounced weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. To translate these diagnostics into scalable training signals, we introduce SciDocIR, a typed evidence-graph representation that preserves scientific document objects, layout and cross-reference relations, and provenance. Building on SciDocIR, we construct SciDocDataset, comprising approximately 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks. Together, SciDocBench, SciDocIR, and SciDocDataset form an evaluation-to-training framework for diagnosing and improving scientific-document assistants. The project page is available at https://github.com/InternLM/SciDocBench.

Related papers

• SciForge: An AI-Native, Multimodal Workbench for Scientific Discovery [48.99744520105086]
  SciForge is a multimodal research-native AI workbench.<n>It reserves the graphical interface for human judgment while search, parsing, model routing, workflow execution, plotting, writing, and presentation generation run as modular agent-accessible services.
  arXiv  Detail & Related papers  (2026-07-17T15:13:03Z)
• SciMDR: Benchmarking and Advancing Scientific Multimodal Document Reasoning [59.93568326270048]
  We introduce the synthesize synthesize framework, which generates faithful, isolated QA pairs on focused segments.<n>We construct SciMDR, a large-scale training dataset for crossmodal comprehension.<n>Experiments demonstrate that SciMDR models achieve significant improvements across multiple scientific QA benchmarks.
  arXiv  Detail & Related papers  (2026-03-12T17:57:52Z)
• ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference [19.363580418167114]
  We formalize multi-document scientific inference, a new task that extracts and aligns motivation, methodology, and experimental results across related papers.<n>We present ResearchPulse, an agent-based framework that integrates instruction planning, scientific content extraction, and structured visualization.<n>To support this task, we introduce ResearchPulse-Bench, a citation-aware benchmark of annotated paper clusters.
  arXiv  Detail & Related papers  (2025-09-03T15:45:03Z)
• SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation [2.985620880452744]
  SciSage is a multi-agent framework employing a reflect-when-you-write paradigm.<n>It critically evaluates drafts at outline, section, and document levels, collaborating with specialized agents for query interpretation, content retrieval, and refinement.<n>We also release SurveyScope, a benchmark of 46 high-impact papers ( 2020-2025) across 11 computer science domains.
  arXiv  Detail & Related papers  (2025-06-15T02:23:47Z)
• OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations [22.336858733121158]
  We introduce OmniDocBench, a novel benchmark featuring high-quality annotations across nine document sources.<n>We conduct a thorough evaluation of both pipeline-based methods and end-to-end vision-language models.
  arXiv  Detail & Related papers  (2024-12-10T16:05:56Z)
• BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks [57.589795399265945]
  We introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks.<n>We also introduce BigDocs-Bench, a benchmark suite with 10 novel tasks.<n>Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o.
  arXiv  Detail & Related papers  (2024-12-05T21:41:20Z)
• DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models [63.466265039007816]
  We present DocGenome, a structured document benchmark constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community. We conduct extensive experiments to demonstrate the advantages of DocGenome and objectively evaluate the performance of large models on our benchmark.
  arXiv  Detail & Related papers  (2024-06-17T15:13:52Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [97.31347312130119]
  SciRIFF (Scientific Resource for Instruction-Following and Finetuning) is a dataset of 137K instruction-following instances for training and evaluation, covering 54 tasks.<n>These tasks span five core scientific literature understanding capabilities: information extraction, summarization, question answering, claim verification, and classification.<n> SciRIFF is unique in being entirely expert-written, high-quality instruction-following dataset for extracting and synthesizing information from research literature across diverse scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• SciRepEval: A Multi-Format Benchmark for Scientific Document Representations [52.01865318382197]
  We introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats. A new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance.
  arXiv  Detail & Related papers  (2022-11-23T21:25:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.