A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation

• URL: http://arxiv.org/abs/2607.28987v1
• Date: Fri, 31 Jul 2026 03:30:25 GMT
• Title: A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation
• Abstract summary: Vision-language models can generate PlantUML from diagram images, but prompt-based use treats this as image-to-text generation rather than structured model generation.<n>We adapt a vision-language model for handwritten-to-PlantUML generation using super-vised fine-tuning followed by Group Relative Policy.<n>Emerging results show that the adapted model improves compilability and conversion quality over the untuned open model and one proprietary baseline, while remaining competitive with a stronger proprietary baseline on class diagrams.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Handwritten UML sketches are common in early software design, but turning them into structured, analysable modelling artefacts still requires manual reconstruction. Vision-language models can generate PlantUML from diagram images, but prompt-based use treats this as image-to-text generation rather than structured model generation. We investigate formalism-aware rewards: feedback signals derived from analysable model representations rather than surface text. In a worked example, we adapt a vision-language model for handwritten UML-to-PlantUML generation using super-vised fine-tuning followed by Group Relative Policy Optimisation. Generated PlantUML is compared against target representations, using XMI for class diagrams and control-flow graphs for activity diagrams. Emerging results show that the adapted model improves compilability and conversion quality over the untuned open model and one proprietary baseline, while remaining competitive with a stronger proprietary baseline on class diagrams. The added benefit of the reward-guided stage remains open on the current held-out set. Error analysis and metric-validity results show that modelling acceptability is only partially captured, motivating rewards and evaluations that combine model analysis with human judgement.

Related papers

• Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling [1.7587442088965224]
  Large language models (LLMs) are increasingly applied to diagram-based software and data modelling.<n>Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed.<n>This systematic literature review analyses 64 studies published between 2023 and 2025.
  arXiv  Detail & Related papers  (2026-07-28T06:41:32Z)
• GloTok: Global Perspective Tokenizer for Image Reconstruction and Generation [51.95701097588426]
  We introduce a Global Perspective Tokenizer (GloTok) to model a more uniform semantic distribution of tokenized features.<n>A residual learning module is proposed to recover the fine-grained details to minimize the reconstruction error caused by quantization.<n>Experiments on the standard ImageNet-1k benchmark clearly show that our proposed method achieves state-of-the-art reconstruction performance and generation quality.
  arXiv  Detail & Related papers  (2025-11-18T06:40:26Z)
• GRAM: A Generative Foundation Reward Model for Reward Generalization [48.63394690265176]
  We develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning.<n>This model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning.
  arXiv  Detail & Related papers  (2025-06-17T04:34:27Z)
• Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation [61.31036260686349]
  We propose a novel prompt optimization framework, designed to rephrase a simple user prompt into a sophisticated prompt to a text-to-image model.<n> Specifically, we employ the large vision language models (LVLMs) as the solver to rewrite the user prompt, and concurrently, employ LVLMs as a reward model to score the aesthetics and alignment of the images generated by the optimized prompt.<n>Instead of laborious human feedback, we exploit the prior knowledge of the LVLM to provide rewards, i.e., AI feedback.
  arXiv  Detail & Related papers  (2025-05-22T15:05:07Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next-token prediction is the fundamental principle for training large language models (LLMs)<n>We introduce R1-SGG, a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset.<n>We design a set of graph-centric rewards, including three recall-based variants -- Hard Recall, Hard Recall+Relax, and Soft Recall.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models [0.41942958779358674]
  This paper proposes a new approach to generate code using a large multimodal language model automatically.<n> domain-adapted MM-LLMs perform for code generation automation, whereby at the best model, it achieved BLEU and SSIM scores of 0.779 and 0.942 on sequence diagrams.
  arXiv  Detail & Related papers  (2025-03-15T23:20:26Z)
• Latent Thought Models with Variational Bayes Inference-Time Computation [52.63299874322121]
  Latent Thought Models (LTMs) incorporate explicit latent thought vectors that follow an explicit prior model in latent space.<n>LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models.
  arXiv  Detail & Related papers  (2025-02-03T17:50:34Z)
• Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning [78.72226641279863]
  Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. Our research explores task-specific model pruning to inform decisions about designing SMoE architectures. We introduce an adaptive task-aware pruning technique UNCURL to reduce the number of experts per MoE layer in an offline manner post-training.
  arXiv  Detail & Related papers  (2024-09-02T22:35:03Z)
• From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs [1.961305559606562]
  In software engineering processes, systems are first specified using a modeling language. Large Language Models (LLM) are used to generate the formal representation of (UML) models from a given drawing. More specifically, we have evaluated the capabilities of different LLMs to convert images of class diagrams into the actual models represented in the images.
  arXiv  Detail & Related papers  (2024-04-17T13:33:11Z)
• Model Generation with LLMs: From Requirements to UML Sequence Diagrams [9.114284818139069]
  This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., sequence diagrams, from NL requirements. We examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges.
  arXiv  Detail & Related papers  (2024-04-09T15:07:25Z)
• Multi-modal Auto-regressive Modeling via Visual Words [96.25078866446053]
  We propose the concept of visual tokens, which maps the visual features to probability distributions over Large Multi-modal Models' vocabulary. We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.
  arXiv  Detail & Related papers  (2024-03-12T14:58:52Z)
• Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizations [1.709620026135923]
  Large language models (LLM) have become an interesting option for supporting generative tasks related to visualization. This paper copes with the problem of modeling the evaluation of a generated visualization through an LLM. We propose a theoretical evaluation stack, EvaLLM, that decomposes the evaluation effort in its atomic components.
  arXiv  Detail & Related papers  (2024-02-03T14:28:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.