Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations

• URL: http://arxiv.org/abs/2605.25101v1
• Date: Sun, 24 May 2026 14:30:56 GMT
• Title: Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations
• Abstract summary: In industrial domains, Functional Mock-up Interface (FMI) is used to exchange simulation models as Functional Mock-up Units (FMUs)<n> deriving effective test oracles for these simulation models remains challenging due to the absence of explicit expected outputs.<n>We propose an LLM-powered multi-agent workflow for specification-based metamorphic testing of FMU-based simulation models.
• Score: 1.58310730488265
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In many industrial domains, the Functional Mock-up Interface (FMI) is used to exchange simulation models as Functional Mock-up Units (FMUs) across different partners using various modelling tools. This opens up the possibilities for simulation-based verification and validation using FMUs for ensuring reliable system behaviour. However, deriving effective test oracles for these simulation models remains challenging due to the absence of explicit expected outputs. This limits the applicability of conventional testing approaches, which require access to the internal workings of the systems. Metamorphic testing (MT) addresses this limitation by leveraging metamorphic relations (MRs), but extracting such relations from specifications remains largely a manual and error-prone process. To address this challenge, we propose an LLM-powered multi-agent workflow for specification-based metamorphic testing of FMU-based simulation models. The approach takes functional and interface specifications as input and orchestrates multiple agents to extract requirements and derive MRs. These MRs are expressed using Given-When-Then patterns to structure input conditions (Given), transformations (When), and expected output behaviours (Then). These relations are then used to generate metamorphic test cases, execute simulations, and evaluate output consistency across multiple sessions. We evaluate the approach on a Lube Oil Cooling system FMU, demonstrating its ability to automatically generate meaningful MRs and corresponding test cases. Preliminary results indicate that the proposed workflow can effectively support the systematic verification and validation of dynamic simulation models by reducing manual effort and improving test generation.

Related papers

• Generative Artificial Intelligence (GenAI) to convert images of queuing networks into verifiable simulation models: an open-weight LLM workflow approach [0.0]
  Sketch2DES is a sketch-to-simulation workflow that converts diagrammatic representations of queuing networks into verifiable simulation models.<n>We evaluate the approach on eight queuing-network diagrams of varying complexity.
  arXiv  Detail & Related papers  (2026-07-27T10:50:37Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Using Large Language Models for Black-Box Testing of FMU-Based Simulations [1.6624933615451842]
  The goal is to reduce the manual effort in defining test scenarios for dynamic simulation models.<n>The approach takes the functional and interface specifications of an FMU as input, and prompts an LLM to generate structured scenario goals.<n>It produces human-readable logs and plots that show statistics for each scenario with overlays, aggregate pass rates, and per-goal outcomes.
  arXiv  Detail & Related papers  (2026-04-28T13:48:24Z)
• SimMOF: AI agent for Automated MOF Simulations [1.5280683380195148]
  Metal-organic frameworks (MOFs) offer a vast design space.<n>SimMOF automates a large language model based multi agent framework that translates end-to-end MOF simulation from natural language queries.<n>SimMOF user requests into dependency aware plans, generates runnable inputs, orchestrates multiple agents to execute simulations, and summarizes results aligned to the user query.
  arXiv  Detail & Related papers  (2026-03-31T02:08:50Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Revisit Mixture Models for Multi-Agent Simulation: Experimental Study within a Unified Framework [19.558523263211942]
  In multi-agent simulation, the primary challenges include behavioral multimodality and closed-loop distributional shifts.<n>In this study, we revisit mixture models for generating multimodal agent behaviors, which can cover the mainstream methods.<n>We introduce a closed-loop sample generation approach tailored for mixture models to mitigate distributional shifts.
  arXiv  Detail & Related papers  (2025-01-28T15:26:25Z)
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• MetaOpenFOAM: an LLM-based multi-agent framework for CFD [11.508919041921942]
  MetaOpenFOAM is a novel multi-agent collaborations framework. It aims to complete CFD simulation tasks with only natural language as input. It harnesses the power of MetaGPT's assembly line paradigm.
  arXiv  Detail & Related papers  (2024-07-31T04:01:08Z)
• Towards a Complete Metamorphic Testing Pipeline [56.75969180129005]
  Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT) These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes.
  arXiv  Detail & Related papers  (2023-09-30T10:49:22Z)
• Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL [154.13105285663656]
  A cooperative Multi-A gent R einforcement Learning (MARL) with permutation invariant agents framework has achieved tremendous empirical successes in real-world applications. Unfortunately, the theoretical understanding of this MARL problem is lacking due to the curse of many agents and the limited exploration of the relational reasoning in existing works. We prove that the suboptimality gaps of the model-free and model-based algorithms are independent of and logarithmic in the number of agents respectively, which mitigates the curse of many agents.
  arXiv  Detail & Related papers  (2022-09-20T16:42:59Z)
• Multi-fidelity Hierarchical Neural Processes [79.0284780825048]
  Multi-fidelity surrogate modeling reduces the computational cost by fusing different simulation outputs. We propose Multi-fidelity Hierarchical Neural Processes (MF-HNP), a unified neural latent variable model for multi-fidelity surrogate modeling. We evaluate MF-HNP on epidemiology and climate modeling tasks, achieving competitive performance in terms of accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2022-06-10T04:54:13Z)
• Calibrating Over-Parametrized Simulation Models: A Framework via Eligibility Set [3.862247454265944]
  We develop a framework to develop calibration schemes that satisfy rigorous frequentist statistical guarantees. We demonstrate our methodology on several numerical examples, including an application to calibration of a limit order book market simulator.
  arXiv  Detail & Related papers  (2021-05-27T00:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.