Unlocking UML Class Diagram Understanding in Vision Language Models

• URL: http://arxiv.org/abs/2605.11634v1
• Date: Tue, 12 May 2026 06:57:47 GMT
• Title: Unlocking UML Class Diagram Understanding in Vision Language Models
• Abstract summary: Vision Language Models (VLMs) fall behind in answering questions regard-ing diagrams compared to photos.<n>We present a benchmark for visual question answering based on class diagrams which is both challenging and manageable.<n>A LoRA-based finetune easily outperforms Qwen 3.5 27B in many other benchmarks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Although Vision Language Models (VLMs) have seen tremendous progress across all kinds of use cases, they still fall behind in answering questions regard-ing diagrams compared to photos. Although progress has been made in the area of bar charts, line charts and other diagrams like that there is still few research concerned with other types of diagrams, e.g. in the computer science domain. Our work presents a benchmark for visual question answering based on UML class diagrams which is both challenging and manageable. We further construct a large-scale training dataset with 16.000 image-question-answer triples and show that a LoRA-based finetune easily outperforms Qwen 3.5 27B, which is a recent and well-performing VLM in many other benchmarks.

Related papers

• From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding [0.0]
  Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images.<n>We introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula.<n>We construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences.
  arXiv  Detail & Related papers  (2026-09-01T09:07:31Z)
• Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams [53.28039059333946]
  We build a benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding.<n> Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains.<n>It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering.
  arXiv  Detail & Related papers  (2026-08-12T17:04:13Z)
• How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing [56.60465182650588]
  We introduce three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning.<n>We propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment.<n>We find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models.
  arXiv  Detail & Related papers  (2026-02-02T09:24:45Z)
• More Images, More Problems? A Controlled Analysis of VLM Failure Modes [80.64323947730905]
  Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored.<n>We introduce MIMIC, a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs.
  arXiv  Detail & Related papers  (2026-01-12T18:45:13Z)
• MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning? [21.777853590188688]
  We present MathSight, a university-level multimodal mathematical reasoning benchmark designed to disentangle and quantify the effect of visual input.<n> Experiments on state-of-the-art Vision-Language Models reveal a consistent trend: the contribution of visual information diminishes with increasing problem difficulty.
  arXiv  Detail & Related papers  (2025-11-28T11:55:05Z)
• CHAOS: Chart Analysis with Outlier Samples [31.64244745491319]
  CHAOS is a benchmark to evaluate Multimodal Large Language Models (MLLMs) against chart perturbations.<n>The benchmark includes 13 state-of-the-art MLLMs divided into three groups according to the training scope and data.
  arXiv  Detail & Related papers  (2025-05-22T19:26:49Z)
• RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning [63.599057862999]
  RefChartQA is a novel benchmark that integrates Chart Question Answering (ChartQA) with visual grounding.<n>Our experiments demonstrate that incorporating spatial awareness via grounding improves response accuracy by over 15%.
  arXiv  Detail & Related papers  (2025-03-29T15:50:08Z)
• Systematic teaching of UML and behavioral diagrams [0.0]
  This article discusses the systematic acquisition of skills required for creating diagrams. The more unusual question types are related to images, such as questions about image annotation. All the demonstrated exercises are suitable for both digital and handwritten training or exams.
  arXiv  Detail & Related papers  (2024-10-23T13:18:54Z)
• Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path [53.71787069694794]
  We focus on the graph reasoning ability of Large Language Models (LLMs)<n>We revisit the ability of LLMs on three fundamental graph tasks: graph description translation, graph connectivity, and the shortest-path problem.<n>Our findings suggest that LLMs can fail to understand graph structures through text descriptions and exhibit varying performance for all these fundamental tasks.
  arXiv  Detail & Related papers  (2024-08-18T16:26:39Z)
• ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning [55.22996841790139]
  We benchmark the ability of off-the-shelf Multi-modal Large Language Models (MLLMs) in the chart domain.<n>We construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data.<n>We develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns.
  arXiv  Detail & Related papers  (2024-02-19T14:48:23Z)
• Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization [27.913656283822483]
  Large language models (LLMs) have shown impressive generalization capabilities to unseen tasks. We propose PromptChart, a multimodal few-shot prompting framework with LLMs for chart-related applications. Our experiments on three different chart-related information consumption tasks show that with properly designed prompts LLMs can excel on the benchmarks.
  arXiv  Detail & Related papers  (2023-12-17T05:13:58Z)
• Classification of Reverse-Engineered Class Diagram and Forward-Engineered Class Diagram using Machine Learning [0.0]
  In software industry it is important to know which type of class diagram it is. Which diagram was used in a particular project is an important factor to be known? We propose to solve this problem by using a supervised Machine Learning technique.
  arXiv  Detail & Related papers  (2020-11-14T14:56:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.