DiagramRAG: A Lightweight Framework to Retrieve Scientific Diagram for Figure Generation

• URL: http://arxiv.org/abs/2605.27931v1
• Date: Wed, 27 May 2026 04:03:22 GMT
• Title: DiagramRAG: A Lightweight Framework to Retrieve Scientific Diagram for Figure Generation
• Abstract summary: We introduce DiagramRAG, a lightweight retrieval-augmented framework for sketch-based scientific diagram completion.<n>Given a user sketch, DiagramRAG retrieves reference diagrams that are both semantically relevant to the sketch content and topologically compatible with its structure.<n>Experiments show that DiagramRAG achieves F1-scores of 0.848 and 0.802 on DiagramBank and FigureBench, respectively, and improves generation quality with the best VLM-as-a-Judge score of 7.170.
• Score: 9.701968199439387
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scientific diagrams are essential for communicating complex methodologies in academic papers. A natural way for researchers to specify such diagrams is through rough sketches, where text labels, connectors, and spatial arrangements express early semantic and topological intentions. However, sketches are usually incomplete, making them insufficient for directly producing publication-quality diagrams. Existing sketch-based generation methods mainly reconstruct the sketch itself, while recent text-driven diagram generation frameworks rely on textual semantics and do not fully exploit the topological structure contained in sketches. In this paper, we introduce DiagramRAG, a lightweight retrieval-augmented framework for sketch-based scientific diagram completion. Given a user sketch, DiagramRAG retrieves reference diagrams that are both semantically relevant to the sketch content and topologically compatible with its structure, and uses them to guide downstream diagram generation. To enable efficient structure-aware retrieval, we represent diagrams as knowledge graphs, synthesize sketch variants at different simplification levels, and train an embedding model to align sketches with compatible diagrams in a shared space. The retrieved references further provide content, topology, and visual priors for completing and rendering the final diagram. Experiments show that DiagramRAG achieves F1-scores of 0.848 and 0.802 on DiagramBank and FigureBench, respectively, and improves generation quality with the best VLM-as-a-Judge score of 7.170, while reducing inference latency to 35.48 seconds per sample. Our code and data are available at https://anonymous.4open.science/r/DiagramRAG-A262 and https://huggingface.co/datasets/anonymous-review-a262/DiagramSketch.

Related papers

• From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding [0.0]
  Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images.<n>We introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula.<n>We construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences.
  arXiv  Detail & Related papers  (2026-09-01T09:07:31Z)
• DiagramBank: A Large-scale Dataset of Diagram Design Exemplars with Paper Metadata for Retrieval-Augmented Generation [4.248969416909163]
  We present DiagramBank, a large-scale dataset consisting of 89,422 schematic diagrams curated from top-tier scientific publications.<n>DiagramBank is developed through our automated curation pipeline that extracts figures and corresponding in-text references.<n>We release DiagramBank in a ready-to-index format and provide a retrieval-augmented generation to demonstrate exemplar-conditioned synthesis of teaser figures.
  arXiv  Detail & Related papers  (2026-02-28T04:17:35Z)
• DiagramEval: Evaluating LLM-Generated Diagrams via Graphs [25.040934047462112]
  We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs.<n>We propose DiagramEval, a novel evaluation metric designed to assess the quality of demonstration diagrams generated by large language models.
  arXiv  Detail & Related papers  (2025-10-29T17:56:17Z)
• SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches [54.06877048295693]
  We introduce SketchAgent, a system designed to automate the transformation of hand-drawn sketches into structured diagrams.<n>SketchAgent integrates sketch recognition, symbolic reasoning, and iterative validation to produce semantically coherent and structurally accurate diagrams.<n>By streamlining the diagram generation process, SketchAgent holds great promise for applications in design, education, and engineering.
  arXiv  Detail & Related papers  (2025-08-02T07:22:51Z)
• Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation [79.75818239774952]
  Large language models (LLMs) have demonstrated remarkable capabilities, but still struggle with issues like hallucinations and outdated information.<n>Retrieval-augmented generation (RAG) addresses these issues by grounding LLM outputs in external knowledge with an Information Retrieval (IR) system.<n>We propose Align-GRAG, a novel reasoning-guided dual alignment framework in post-retrieval phrase.
  arXiv  Detail & Related papers  (2025-05-22T05:15:27Z)
• DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning [62.51232333352754]
  Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. We present DiagrammerGPT, a novel two-stage text-to-diagram generation framework. We show that our framework produces more accurate diagrams, outperforming existing T2I models.
  arXiv  Detail & Related papers  (2023-10-18T17:37:10Z)
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval [147.24102408745247]
  We study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels.
  arXiv  Detail & Related papers  (2020-07-29T20:50:25Z)
• Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion [53.31911669146451]
  Human-curated knowledge graphs provide critical supportive information to various natural language processing tasks. These graphs are usually incomplete, urging auto-completion of them. graph embedding approaches, e.g., TransE, learn structured knowledge via representing graph elements into dense embeddings. textual encoding approaches, e.g., KG-BERT, resort to graph triple's text and triple-level contextualized representations.
  arXiv  Detail & Related papers  (2020-04-30T13:50:34Z)
• Bridging Knowledge Graphs to Generate Scene Graphs [49.69377653925448]
  We propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs.
  arXiv  Detail & Related papers  (2020-01-07T23:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.