Assessing UML Models by ChatGPT: Implications for Education

• URL: http://arxiv.org/abs/2412.17200v1
• Date: Mon, 23 Dec 2024 00:28:33 GMT
• Title: Assessing UML Models by ChatGPT: Implications for Education
• Abstract summary: In software engineering (SE) research and practice, is well known as an essential modeling methodology.<n>Recent advancements in generative AI techniques, such as ChatGPT, have paved new ways to automate many SE tasks.<n>This paper aims to investigate the feasibility and effectiveness of ChatGPT in assessing the quality of models.
• Score: 9.11195766839205
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In software engineering (SE) research and practice, UML is well known as an essential modeling methodology for requirements analysis and software modeling in both academia and industry. In particular, fundamental knowledge of UML modeling and practice in creating high-quality UML models are included in SE-relevant courses in the undergraduate programs of many universities. This leads to a time-consuming and labor-intensive task for educators to review and grade a large number of UML models created by the students. Recent advancements in generative AI techniques, such as ChatGPT, have paved new ways to automate many SE tasks. However, current research or tools seldom explore the capabilities of ChatGPT in evaluating the quality of UML models. This paper aims to investigate the feasibility and effectiveness of ChatGPT in assessing the quality of UML use case diagrams, class diagrams, and sequence diagrams. First, 11 evaluation criteria with grading details were proposed for these UML models. Next, a series of experiments were designed and conducted on 40 students' UML modeling reports to explore the performance of ChatGPT in evaluating and grading these UML diagrams. The research findings reveal that ChatGPT performed well in this assessing task because the scores that ChatGPT gives to the UML models are similar to the ones by human experts, and there are three evaluation discrepancies between ChatGPT and human experts, but varying in different evaluation criteria used in different types of UML models.

Related papers

• UML Class Diagram Evaluation and Repair Strategies based on LLMs [7.582062038207369]
  With the development of AI technologies, particularly LLMs, new opportunities for software modeling have emerged.<n>This study conducts experiments on several typical software system cases.
  arXiv  Detail & Related papers  (2026-08-28T19:10:30Z)
• Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams [53.28039059333946]
  We build a benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding.<n> Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains.<n>It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering.
  arXiv  Detail & Related papers  (2026-08-12T17:04:13Z)
• Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling [1.7587442088965224]
  Large language models (LLMs) are increasingly applied to diagram-based software and data modelling.<n>Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed.<n>This systematic literature review analyses 64 studies published between 2023 and 2025.
  arXiv  Detail & Related papers  (2026-07-28T06:41:32Z)
• Enhancing a gamified tool for UML modeling education [2.3449922785286965]
  Gamification has been adopted to make modeling education more appealing.<n>Existing tools focus almost exclusively on class diagrams, leaving support for use cases and other notations unexplored.<n>In 2025, we designed UMLegend, a gamified tool for class diagrams that offered dynamic feedback to help students learn correct modeling practices.<n>This paper describes how we enhanced UMLegend following the results of the experiment so that it can support more modeling languages.
  arXiv  Detail & Related papers  (2026-04-24T09:41:45Z)
• NOMAD: A Multi-Agent LLM System for UML Class Diagram Generation from Natural Language Requirements [20.080985332719383]
  Large Language Models (LLMs) are increasingly utilised in software engineering, yet their ability to generate structured artefacts such as diagrams remains underexplored.<n>In this work we present NOMAD, a cognitively inspired, modular multi-agent framework that decomposes generation into a series of role-specialised subtasks.<n>Each agent handles a distinct modelling activity, such as entity extraction, relationship classification, synthesis diagram, mirroring the goal-directed reasoning processes of an engineer.
  arXiv  Detail & Related papers  (2025-11-27T12:36:25Z)
• Rethinking Machine Unlearning in Image Generation Models [59.697750585491264]
  CatIGMU is a novel hierarchical task categorization framework.<n>EvalIGMU is a comprehensive evaluation framework.<n>We construct DataIGM, a high-quality unlearning dataset.
  arXiv  Detail & Related papers  (2025-06-03T11:25:14Z)
• Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models [0.41942958779358674]
  This paper proposes a new approach to generate code using a large multimodal language model automatically. domain-adapted MM-LLMs perform for code generation automation, whereby at the best model, it achieved BLEU and SSIM scores of 0.779 and 0.942 on sequence diagrams.
  arXiv  Detail & Related papers  (2025-03-15T23:20:26Z)
• Towards Understanding Graphical Perception in Large Multimodal Models [80.44471730672801]
  We leverage the theory of graphical perception to develop an evaluation framework for analyzing gaps in LMMs' perception abilities in charts.<n>We apply our framework to evaluate and diagnose the perception capabilities of state-of-the-art LMMs at three levels (chart, visual element, and pixel)
  arXiv  Detail & Related papers  (2025-03-13T20:13:39Z)
• MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs [97.94579295913606]
  Multimodal Large Language Models (MLLMs) have garnered increased attention from both industry and academia.<n>In the development process, evaluation is critical since it provides intuitive feedback and guidance on improving models.<n>This work aims to offer researchers an easy grasp of how to effectively evaluate MLLMs according to different needs and to inspire better evaluation methods.
  arXiv  Detail & Related papers  (2024-11-22T18:59:54Z)
• How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension [53.6373473053431]
  This work introduces a benchmark to assess large language models' capabilities in graph pattern tasks.<n>We have developed a benchmark that evaluates whether LLMs can understand graph patterns based on either terminological or topological descriptions.<n>Our benchmark encompasses both synthetic and real datasets, and a variety of models, with a total of 11 tasks and 7 models.
  arXiv  Detail & Related papers  (2024-10-04T04:48:33Z)
• OLMES: A Standard for Language Model Evaluations [64.85905119836818]
  We propose OLMES, a practical, open standard for reproducible language model evaluations. We identify and review the varying factors in evaluation practices adopted by the community. OLMES supports meaningful comparisons between smaller base models that require the unnatural "cloze" formulation of multiple-choice questions.
  arXiv  Detail & Related papers  (2024-06-12T17:37:09Z)
• How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts [9.517655899237413]
  GPT-3, Large Language Models (LLMs) have caught the eyes of researchers, practitioners, and educators in the field of software engineering. This paper explores how LLMs can assist novice analysts in creating three types of typical models: use case models, class diagrams, and sequence diagrams.
  arXiv  Detail & Related papers  (2024-04-27T00:38:20Z)
• Assessing GPT-4-Vision's Capabilities in UML-Based Code Generation [0.5789654849162464]
  GPT-4-Vision is a state-of-the-art deep learning model. It can transform Unified Modeling Language (UML) class diagrams into fully operating Java class files.
  arXiv  Detail & Related papers  (2024-04-22T17:21:24Z)
• Model Generation with LLMs: From Requirements to UML Sequence Diagrams [9.114284818139069]
  This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., sequence diagrams, from NL requirements. We examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges.
  arXiv  Detail & Related papers  (2024-04-09T15:07:25Z)
• Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizations [1.709620026135923]
  Large language models (LLM) have become an interesting option for supporting generative tasks related to visualization. This paper copes with the problem of modeling the evaluation of a generated visualization through an LLM. We propose a theoretical evaluation stack, EvaLLM, that decomposes the evaluation effort in its atomic components.
  arXiv  Detail & Related papers  (2024-02-03T14:28:55Z)
• Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels [95.44077384918725]
  We propose to teach large multi-modality models (LMMs) with text-defined rating levels instead of scores. The proposed Q-Align achieves state-of-the-art performance on image quality assessment (IQA), image aesthetic assessment (IAA) and video quality assessment (VQA) tasks.
  arXiv  Detail & Related papers  (2023-12-28T16:10:25Z)
• BLESS: Benchmarking Large Language Models on Sentence Simplification [55.461555829492866]
  We present BLESS, a performance benchmark of the most recent state-of-the-art large language models (LLMs) on the task of text simplification (TS) We assess a total of 44 models, differing in size, architecture, pre-training methods, and accessibility, on three test sets from different domains (Wikipedia, news, and medical) under a few-shot setting. Our evaluation indicates that the best LLMs, despite not being trained on TS, perform comparably with state-of-the-art TS baselines.
  arXiv  Detail & Related papers  (2023-10-24T12:18:17Z)
• Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming [29.65988680948297]
  We explore the application of large language models (LLMs) for in-context student modeling in open-ended learning domains. We introduce a novel framework, LLM for Student Synthesis (LLM-SS), that leverages LLMs for a student's behavior. We instantiate several methods based on LLM-SS framework and evaluate them using an existing benchmark, StudentSyn, for student attempt synthesis in a visual programming domain.
  arXiv  Detail & Related papers  (2023-10-15T12:56:13Z)
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark [53.663757126289795]
  This paper introduces GAOKAO-Bench, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples. With human evaluation, we obtain the converted total score of LLMs, including GPT-4, ChatGPT and ERNIE-Bot. We also use LLMs to grade the subjective questions, and find that model scores achieve a moderate level of consistency with human scores.
  arXiv  Detail & Related papers  (2023-05-21T14:39:28Z)
• Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification [58.720142291102135]
  This case study investigates the task of job classification in a real-world setting. The goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position.
  arXiv  Detail & Related papers  (2023-03-13T14:09:53Z)
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization [101.37439352091612]
  We describe the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. We present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT.
  arXiv  Detail & Related papers  (2022-12-22T19:56:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.