Can LLMs Learn and Apply Multi-Level Modelling Semantics? A First Empirical Study

• URL: http://arxiv.org/abs/2607.13257v1
• Date: Tue, 14 Jul 2026 20:41:43 GMT
• Title: Can LLMs Learn and Apply Multi-Level Modelling Semantics? A First Empirical Study
• Abstract summary: Multi-level modelling (MLM) can directly represent three or more abstraction levels, but this comes at the cost of more complex semantic constraints that model correctness depends on.<n>This paper presents the first empirical study of this question.
• Score: 2.9612857576788447
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Industry 5.0 emphasises human-centric industrial system design, placing additional demands on modelling tools. Multi-level modelling (MLM) can directly represent three or more abstraction levels, but this comes at the cost of more complex semantic constraints that model correctness depends on. Large Language Models (LLMs) have been increasingly studied in model-driven engineering, but this evidence rests entirely on two-level modelling tasks, and whether it generalises to MLM, whose semantics differ in kind, remains untested. This paper presents the first empirical study of this question. We have three commercial LLMs (GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro) generate multi-level models for the MULTI Warehouse Challenge in the SLICER language under six prompting strategies, yielding 90 generated models compared against a manually validated reference using fourteen metrics. Syntactic correctness is within reach, but semantic correctness is only partially achieved, with Instantiation/Specialisation Correctness ranging from 52% to 79%. Models reproduce content stated explicitly in the task text, but rarely complete structure and constraints the text implies without stating. Prompting strategies trade off precision against completeness, and self-checking functions mainly as a rule checker rather than reliably improving alignment with the reference design. Among the three LLMs, Claude shows the most balanced profile. These results clarify the boundaries of current LLM capability for MLM and inform the design of human-centred, AI-assisted modelling workflows for Industry 5.0.

Related papers

• Reliability of Large Language Models for Design Synthesis: An Empirical Study of Variance, Prompt Sensitivity, and Method Scaffolding [2.1843439591862333]
  Large Language Models (LLMs) are increasingly applied to automate software engineering tasks.<n>This study investigates whether LLMs can move beyond diagram translation to perform design synthesis.
  arXiv  Detail & Related papers  (2026-04-01T13:05:09Z)
• Class Model Generation from Requirements using Large Language Models [5.685497917524985]
  Large Language Models (LLMs) can generate class diagrams from natural language requirements automatically.<n>This paper investigates the capabilities of state-of-the-art LLMs, including GPT-5, Claude Sonnet 4.0, Gemini 2.5 Flash Thinking, and Llama-3.1-8-B-Instruct.
  arXiv  Detail & Related papers  (2026-03-10T02:20:35Z)
• One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking [7.856998585396422]
  Large language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines.<n>We propose textbfmulti-task learning (MTL) as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly.
  arXiv  Detail & Related papers  (2026-01-16T13:44:25Z)
• Can large language models assist choice modelling? Insights into prompting strategies and current models capabilities [0.0]
  Large Language Models (LLMs) are widely used to support various disciplines, yet their potential in choice modelling remains relatively unexplored.<n>This work examines the potential of LLMs as assistive agents in the specification and, where technically feasible, estimation of Multinomial Logit models.
  arXiv  Detail & Related papers  (2025-07-29T13:24:44Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• MM-R$^3$: On (In-)Consistency of Vision-Language Models (VLMs) [26.475993408532304]
  We analyze performance of SoTA Vision Language Models on three tasks: Question Rephrasing, Image Restyling, and Context Reasoning.<n>Our analysis reveals that consistency does not always align with accuracy, indicating that models with higher accuracy are not necessarily more consistent, and vice versa.<n>We propose a simple yet effective mitigation strategy in the form of an adapter module trained to minimize inconsistency across prompts.
  arXiv  Detail & Related papers  (2024-10-07T06:36:55Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• Mixed Distillation Helps Smaller Language Model Better Reasoning [27.934081882868902]
  We introduce Mixed Distillation (MD) framework, which capitalizes on the strengths of Program of Thought (PoT) and Chain of Thought (CoT) capabilities within large language models (LLMs) Our experimental results show that MD significantly enhances the single-path and multi-path reasoning ability of smaller models in various tasks.
  arXiv  Detail & Related papers  (2023-12-17T14:28:28Z)
• Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning [79.53130089003986]
  Large Language Models (LLMs) have become a feasible solution for handling tasks in various domains. In this paper, we introduce how to fine-tune a LLM model that can be privately deployed for content moderation.
  arXiv  Detail & Related papers  (2023-10-05T09:09:44Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)
• Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification [58.720142291102135]
  This case study investigates the task of job classification in a real-world setting. The goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position.
  arXiv  Detail & Related papers  (2023-03-13T14:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.