Benchmarking and Evaluating VLMs for Software Architecture Diagram Understanding

• URL: http://arxiv.org/abs/2604.04009v1
• Date: Sun, 05 Apr 2026 07:54:18 GMT
• Title: Benchmarking and Evaluating VLMs for Software Architecture Diagram Understanding
• Abstract summary: Software architecture diagrams are important design artifacts for communicating system structure, behavior, and data organization throughout the software development lifecycle.<n>We present SADU, a benchmark for Software Architecture Diagram Understanding that evaluates modern vision-language models (VLMs) on architecture diagrams as structured software engineering artifacts rather than generic images.
• Score: 6.7195738068448625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Software architecture diagrams are important design artifacts for communicating system structure, behavior, and data organization throughout the software development lifecycle. Although recent progress in large language models has substantially advanced code-centric software engineering tasks such as code generation, testing, and maintenance, the ability of modern vision-language models (VLMs) to understand software architecture diagrams remains underexplored. To address this gap, we present SADU, a benchmark for Software Architecture Diagram Understanding that evaluates VLMs on architecture diagrams as structured software engineering artifacts rather than generic images. SADU contains 154 carefully curated diagrams spanning behavioral, structural, and ER diagrams, paired with structured annotations and 2,431 question-answer tasks covering counting and retrieval reasoning. We evaluate 11 state-of-the-art VLMs from the Gemini, Claude, GPT, and Qwen families. Our results show that software architecture diagram understanding remains challenging for current models: the best-performing model gemini-3-flash-preview achieves only 70.18\% accuracy, while gpt-4o-mini only achieves 17.77\% accuracy. The results further reveal the weaknesses in diagram reasoning and visual relation grounding, highlighting a gap between current VLMs and the needs of design-stage software engineering. SADU provides a foundation for future research on diagram-aware AI systems and more faithful AI-assisted software engineering workflows.

Related papers

• Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation [8.867061253116367]
  Large Language Models (LLMs) have demonstrated significant potential in automating software engineering tasks.<n> Generating software architecture designs from requirement documents is a crucial step in software development.<n>R2ABench is a novel benchmark comprising diverse real-world software projects paired with comprehensive Product Requirements Documents (PRDs) and expert-curated PlantUML reference diagrams.
  arXiv  Detail & Related papers  (2026-04-08T04:58:36Z)
• How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing [56.60465182650588]
  We introduce three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning.<n>We propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment.<n>We find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models.
  arXiv  Detail & Related papers  (2026-02-02T09:24:45Z)
• Hierarchical Process Reward Models are Symbolic Vision Learners [56.94353087007494]
  Symbolic computer vision represents diagrams through explicit logical rules and structured representations, enabling interpretable understanding in machine vision.<n>This requires fundamentally different learning paradigms from pixel-based visual models.<n>We propose a novel self-supervised auto-encoder that encodes diagrams into primitives and decodes them through our executable engine to reconstruct input diagrams.
  arXiv  Detail & Related papers  (2025-12-02T18:46:40Z)
• Generating Software Architecture Description from Source Code using Reverse Engineering and Large Language Model [2.6126272668390373]
  Software Architecture Descriptions (SADs) are essential for managing the inherent complexity of modern software systems.<n>SADs are often missing, outdated, or poorly aligned with the system's actual implementation.<n>We propose a semi-automated generation of SADs from source code by integrating reverse engineering (RE) techniques with a Large Language Model (LLM)
  arXiv  Detail & Related papers  (2025-11-07T11:35:46Z)
• Software Architecture Meets LLMs: A Systematic Literature Review [4.28281840272851]
  We present a systematic literature review on the use of Large Language Models in software architecture.<n>Our findings show that while LLMs are increasingly applied to a variety of software architecture tasks, some areas, such as generating source code from architectural design, remain underexplored.
  arXiv  Detail & Related papers  (2025-05-22T14:00:29Z)
• Assessing LLMs for Front-end Software Architecture Knowledge [0.0]
  Large Language Models (LLMs) have demonstrated significant promise in automating software development tasks.<n>This study investigates the capabilities of an LLM in understanding, reproducing, and generating structures within the VIPER architecture.<n> Experimental results, using ChatGPT 4 Turbo 2024-04-09, reveal that the LLM excelled in higher-order tasks like evaluating and creating, but faced challenges with lower-order tasks requiring precise retrieval of architectural details.
  arXiv  Detail & Related papers  (2025-02-26T19:33:35Z)
• RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph [63.87660059104077]
  We present RepoGraph, a plug-in module that manages a repository-level structure for modern AI software engineering solutions.<n>RepoGraph substantially boosts the performance of all systems, leading to a new state-of-the-art among open-source frameworks.
  arXiv  Detail & Related papers  (2024-10-03T05:45:26Z)
• MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine [85.80851893886161]
  We propose MAVIS, a MAthematical VISual instruction tuning pipeline for MLLMs, featuring an automatic data engine to efficiently create mathematical visual datasets. We use MAVIS-Caption to fine-tune a math-specific vision encoder (CLIP-Math) through contrastive learning, tailored for improved diagram visual encoding. Third, we adopt MAVIS-Instruct to perform the instruction tuning for robust problem-solving skills, and term the resulting model as MAVIS-7B.
  arXiv  Detail & Related papers  (2024-07-11T17:59:47Z)
• Semantic Code Graph -- an information model to facilitate software comprehension [0.0]
  There is an increasing need to accelerate the code comprehension process to facilitate maintenance and reduce associated costs. While a variety of code structure models already exist, there is a surprising lack of models that closely represent the source code. We propose the Semantic Code Graph (SCG), an information model that offers a detailed abstract representation of code dependencies.
  arXiv  Detail & Related papers  (2023-10-03T15:09:49Z)
• Enhancing Architecture Frameworks by Including Modern Stakeholders and their Views/Viewpoints [48.87872564630711]
  The stakeholders with data science and Machine Learning related concerns, such as data scientists and data engineers, are yet to be included in existing architecture frameworks.<n>We surveyed 61 subject matter experts from over 25 organizations in 10 countries.
  arXiv  Detail & Related papers  (2023-08-09T21:54:34Z)
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation [57.92972327649165]
  This work explores a deep learning approach to automatically learn the insecure patterns from code corpora. Because code naturally admits graph structures with parsing, we develop a novel graph neural network (GNN) to exploit both the semantic context and structural regularity of a program.
  arXiv  Detail & Related papers  (2021-09-07T21:24:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.