SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces

• URL: http://arxiv.org/abs/2609.00018v1
• Date: Thu, 20 Aug 2026 02:21:42 GMT
• Title: SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces
• Abstract summary: There is currently no public dataset that pairs specific figures with captions, context, questions, answers, and step-by-step reasoning.<n>We present textbfSCAFFOLD, a large-scale structured dataset of computer science research figures.
• Score: 0.23332469289621782
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Computer science papers rely heavily on diagrams: architecture drawings, system flowcharts, and pipeline schematics that often carry more information than the text around them. There is currently no public dataset that pairs this specific kind of figure with captions, context, questions, answers, and step-by-step reasoning, which is exactly what is needed to train a vision-language model to understand them. We present \textbf{SCAFFOLD}\footnote{https://github.com/theranjitraut/scaffold}, a large-scale structured dataset of computer science research figures with diagram QA and Chain-of-Thought reasoning traces. This dataset consists of (image, caption, context, question-answer, chain-of-thought) tuples from arXiv computer science papers prepared using layout detection and PDF parsing, with an AI-assisted question-generation step. The resulting large-sized SCAFFOLD-157K dataset spans 3,058 papers with 29,887 figures (157,387 pairs), a medium-sized SCAFFOLD-37K dataset (36,797 pairs), and a small-sized SCAFFOLD-12K dataset (12,000 pairs). We used SCAFFOLD-12K for baseline experiments on Qwen2.5-VL-3B-Instruct.

Related papers

• DiagramBank: A Large-scale Dataset of Diagram Design Exemplars with Paper Metadata for Retrieval-Augmented Generation [4.248969416909163]
  We present DiagramBank, a large-scale dataset consisting of 89,422 schematic diagrams curated from top-tier scientific publications.<n>DiagramBank is developed through our automated curation pipeline that extracts figures and corresponding in-text references.<n>We release DiagramBank in a ready-to-index format and provide a retrieval-augmented generation to demonstrate exemplar-conditioned synthesis of teaser figures.
  arXiv  Detail & Related papers  (2026-02-28T04:17:35Z)
• ScrapeGraphAI-100k: A Large-Scale Dataset for LLM-Based Web Information Extraction [0.0]
  We introduce ScrapeGraphAI-100k, a large-scale dataset of real-world LLM extraction events.<n>Starting from 9M events, we deduplicate and balance by schema to produce 93,695 examples spanning diverse domains.<n>We characterize the datasets structural diversity and its failure modes as schema complexity.
  arXiv  Detail & Related papers  (2026-02-16T20:56:59Z)
• SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers [43.18330795060871]
  SPIQA is a dataset specifically designed to interpret complex figures and tables within the context of scientific research articles.<n>We employ automatic and manual curation to create the dataset.<n> SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits.
  arXiv  Detail & Related papers  (2024-07-12T16:37:59Z)
• mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model [73.38800189095173]
  This work focuses on strengthening the multi-modal diagram analysis ability of Multimodal LLMs. By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper. M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes.
  arXiv  Detail & Related papers  (2023-11-30T04:43:26Z)
• Interactive Distillation of Large Single-Topic Corpora of Scientific Papers [1.2954493726326113]
  A more robust but time-consuming approach is to build the dataset constructively in which a subject matter expert handpicks documents. Here we showcase a new tool, based on machine learning, for constructively generating targeted datasets of scientific literature.
  arXiv  Detail & Related papers  (2023-09-19T17:18:36Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• Why do tree-based models still outperform deep learning on tabular data? [0.0]
  We show that tree-based models remain state-of-the-art on medium-sized data. We conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs)
  arXiv  Detail & Related papers  (2022-07-18T08:36:08Z)
• Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search [96.31315520244605]
  Arch-Graph is a transferable NAS method that predicts task-specific optimal architectures. We show Arch-Graph's transferability and high sample efficiency across numerous tasks. It is able to find top 0.16% and 0.29% architectures on average on two search spaces under the budget of only 50 models.
  arXiv  Detail & Related papers  (2022-04-12T16:46:06Z)
• Cherry-Picking Gradients: Learning Low-Rank Embeddings of Visual Data via Differentiable Cross-Approximation [53.95297550117153]
  We propose an end-to-end trainable framework that processes large-scale visual data tensors by looking emphat a fraction of their entries only. The proposed approach is particularly useful for large-scale multidimensional grid data, and for tasks that require context over a large receptive field.
  arXiv  Detail & Related papers  (2021-05-29T08:39:57Z)
• A Graph Representation of Semi-structured Data for Web Question Answering [96.46484690047491]
  We propose a novel graph representation of Web tables and lists based on a systematic categorization of the components in semi-structured data as well as their relations. Our method improves F1 score by 3.90 points over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2020-10-14T04:01:54Z)
• ENT-DESC: Entity Description Generation by Exploring Knowledge Graph [53.03778194567752]
  In practice, the input knowledge could be more than enough, since the output description may only cover the most significant knowledge. We introduce a large-scale and challenging dataset to facilitate the study of such a practical scenario in KG-to-text. We propose a multi-graph structure that is able to represent the original graph information more comprehensively.
  arXiv  Detail & Related papers  (2020-04-30T14:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.