Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

• URL: http://arxiv.org/abs/2608.12262v1
• Date: Wed, 12 Aug 2026 17:04:13 GMT
• Title: Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
• Abstract summary: We build a benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding.<n> Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains.<n>It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering.
• Score: 53.28039059333946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.

Related papers

• Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities [0.0]
  We introduce Math-Vision Diagrams, the first benchmark specifically designed to evaluate Large Language Models (LLMs)<n>We select a subset of 2920 images out of 3040 from high-quality competition problems with essential visual context.<n>All code, data, curation pipeline, and evaluation scripts will be fully open-sourced.
  arXiv  Detail & Related papers  (2026-08-09T23:58:01Z)
• Unlocking UML Class Diagram Understanding in Vision Language Models [0.0]
  Vision Language Models (VLMs) fall behind in answering questions regard-ing diagrams compared to photos.<n>We present a benchmark for visual question answering based on class diagrams which is both challenging and manageable.<n>A LoRA-based finetune easily outperforms Qwen 3.5 27B in many other benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T06:57:47Z)
• GraphCogent: Mitigating LLMs' Working Memory Constraints via Multi-Agent Collaboration in Complex Graph Understanding [13.356521655409422]
  Large language models (LLMs) show promising performance on small-scale graph reasoning tasks but fail when handling real-world graphs with complex queries.<n>We propose GraphCogent, a collaborative agent framework that decomposes graph reasoning into specialized cognitive processes: sense, buffer, and execute.
  arXiv  Detail & Related papers  (2025-08-17T14:28:38Z)
• Distill Visual Chart Reasoning Ability from LLMs to MLLMs [64.32993770646165]
  Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs)<n>We propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs.<n>ReachQA is a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and reasoning abilities of MLLMs.
  arXiv  Detail & Related papers  (2024-10-24T14:50:42Z)
• Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models [70.03602551880526]
  We introduce ProGraph, a benchmark for large language models (LLMs) to process graphs.<n>Our findings reveal that the performance of current LLMs is unsatisfactory, with the best model achieving only 36% accuracy.<n>We propose LLM4Graph datasets, which include crawled documents and auto-generated codes based on 6 widely used graph libraries.
  arXiv  Detail & Related papers  (2024-09-29T11:38:45Z)
• Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs [60.71360240206726]
  Large language models (LLMs) suffer from hallucinations, especially on knowledge-intensive tasks. Existing works propose to augment LLMs with individual text units retrieved from external knowledge corpora. We propose a framework called Graph Chain-of-thought (Graph-CoT) to augment LLMs with graphs by encouraging LLMs to reason on the graph iteratively.
  arXiv  Detail & Related papers  (2024-04-10T15:41:53Z)
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? [99.0305256706604]
  We introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning.
  arXiv  Detail & Related papers  (2024-03-21T17:59:50Z)
• mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model [73.38800189095173]
  This work focuses on strengthening the multi-modal diagram analysis ability of Multimodal LLMs. By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper. M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes.
  arXiv  Detail & Related papers  (2023-11-30T04:43:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.