From Text to DSL: Evaluating Grammar-Based Model Generation Using Open LLMs

• URL: http://arxiv.org/abs/2605.15865v1
• Date: Fri, 15 May 2026 11:33:48 GMT
• Title: From Text to DSL: Evaluating Grammar-Based Model Generation Using Open LLMs
• Abstract summary: Large Language Models (LLMs) have shown increasing potential in automating model-driven software engineering tasks.<n>We evaluate whether open-source LLMs of varying sizes can generate DSL-conformant models using only few-shot prompting.
• Score: 2.5576448583538514
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have shown increasing potential in automating model-driven software engineering tasks, particularly in generating models conforming to Domain Specific Languages (DSLs) from natural language. While most existing approaches rely on large proprietary models, their high cost and limited deployability hinder broader adoption. In this paper, we evaluate whether open-source LLMs of varying sizes (0.5B to 32B parameters) can generate DSL-conformant models using only few-shot prompting, without any fine-tuning. Our evaluation focuses on key model-driven engineering (MDE) requirements, including syntactic validity, semantic completeness, and inter-model reference consistency. We extend our prior work by moving from generating user interface models (referred to as "UI models" in this paper) over fixed, predefined data schemas ("data models") to generating both the UI and data models entirely from scratch. This shift serves two purposes: first, it highlights the LLM's ability to infer domain-specific relationships and maintain consistency across multiple interconnected models; second, it allows us to generalize earlier findings by testing DSL generation across models of different natures and structural roles. Our structured evaluation combines automatic parsing and expert feedback across 39 LLMs, revealing that several compact models (e.g., \texttt{gemma3:12b}, \texttt{mistral:7b-instruct}) approach or match the quality of much larger models. These findings demonstrate the feasibility of using smaller, open-source LLMs for grammar-conformant DSL generation in MDE workflows, offering a cost-effective and deployable alternative to closed LLMs.

Related papers

• Model-Driven Discipline for Multi-Agent LLMs: Requirement-to-Verification Generation of Traceable System Models [62.62160094849002]
  RADIANT is an engineering methodology that combines MDE with Multi-Agent Large Language Models.<n>It automatically generates heterogeneous models across engineering phases.<n>It provides exact, automated change-impact analysis.
  arXiv  Detail & Related papers  (2026-07-18T08:50:55Z)
• M*: A Modular, Extensible, Serving System for Multimodal Models [62.77975969000349]
  We present M*, a universal serving system for efficient serving of composite AI models.<n>We show how it can concisely capture composite models from a broad range of families.<n>M* also outperforms the V-JEPA 2-AC rollout baseline for robotic planning by up to 12.5x.
  arXiv  Detail & Related papers  (2026-06-10T21:22:22Z)
• Accurate and Consistent Graph Model Generation from Text with Large Language Models [1.9049294570026933]
  Graph model generation from natural language description is an important task with many applications in software engineering.<n>With the rise of large language models (LLMs), there is a growing interest in using LLMs for graph model generation.<n>We propose a novel abstraction-concretization framework that enhances the consistency and quality of generated graph models.
  arXiv  Detail & Related papers  (2025-08-01T01:52:25Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization [86.8133939108057]
  We propose AdaMMS, a novel model merging method tailored for heterogeneous MLLMs.<n>Our method tackles the challenges in three steps: mapping, merging and searching.<n>As the first model merging method capable of merging heterogeneous MLLMs without labeled data, AdaMMS outperforms previous model merging methods on various vision-language benchmarks.
  arXiv  Detail & Related papers  (2025-03-31T05:13:02Z)
• LLM-enabled Instance Model Generation [4.52634430160579]
  This work explores the generation of instance models using large language models (LLMs)<n>We propose a two-step approach: first, using LLMs to produce a simplified structured output containing all necessary instance model information, and then compiling this intermediate representation into a valid XMI file.<n>Results show that the proposed method significantly improves the usability of LLMs for instance model generation tasks.
  arXiv  Detail & Related papers  (2025-03-28T16:34:29Z)
• Chunk-Distilled Language Modeling [25.238256586953487]
  Chunk-Distilled Language Modeling (CD-LM) is an approach to text generation that addresses two challenges in current large language models (LLMs)<n>Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step.
  arXiv  Detail & Related papers  (2024-12-31T08:32:15Z)
• A Model Is Not Built By A Single Prompt: LLM-Based Domain Modeling With Question Decomposition [4.123601037699469]
  In real-world domain modeling, engineers usually decompose complex tasks into easily solvable sub-tasks. We propose an LLM-based domain modeling approach via question decomposition, similar to developer's modeling process. Preliminary results show that our approach outperforms the single-prompt-based prompt.
  arXiv  Detail & Related papers  (2024-10-13T14:28:04Z)
• UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback [21.858896845159208]
  Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expensive human feedback or distilling a proprietary model. Our method starts with an existing LLM and iteratively produces improved models by self-generating a large synthetic dataset.
  arXiv  Detail & Related papers  (2024-06-11T21:53:46Z)
• LLM Augmented LLMs: Expanding Capabilities through Composition [56.40953749310957]
  CALM -- Composition to Augment Language Models -- introduces cross-attention between models to compose their representations and enable new capabilities. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13% on tasks like translation into English. When PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40% over the base model for code generation and explanation tasks.
  arXiv  Detail & Related papers  (2024-01-04T18:53:01Z)
• Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning [79.53130089003986]
  Large Language Models (LLMs) have become a feasible solution for handling tasks in various domains. In this paper, we introduce how to fine-tune a LLM model that can be privately deployed for content moderation.
  arXiv  Detail & Related papers  (2023-10-05T09:09:44Z)
• Prompt2Model: Generating Deployable Models from Natural Language Instructions [74.19816829003729]
  Large language models (LLMs) enable system builders to create competent NLP systems through prompting. In other ways, LLMs are a step backward from traditional special-purpose NLP models. We propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs.
  arXiv  Detail & Related papers  (2023-08-23T17:28:21Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [47.432215933099016]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.<n>This creates a barrier to fusing knowledge across individual models to yield a better single model.<n>We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.