Using Large Language Models for Black-Box Testing of FMU-Based Simulations

• URL: http://arxiv.org/abs/2604.25650v1
• Date: Tue, 28 Apr 2026 13:48:24 GMT
• Title: Using Large Language Models for Black-Box Testing of FMU-Based Simulations
• Abstract summary: The goal is to reduce the manual effort in defining test scenarios for dynamic simulation models.<n>The approach takes the functional and interface specifications of an FMU as input, and prompts an LLM to generate structured scenario goals.<n>It produces human-readable logs and plots that show statistics for each scenario with overlays, aggregate pass rates, and per-goal outcomes.
• Score: 1.6624933615451842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a human in the loop approach for black-box testing of Functional Mock-up Units (FMUs) using Large Language Models (LLMs). The goal is to reduce the manual effort in defining test scenarios for dynamic simulation models and to improve the interpretability of results. The approach takes the functional and interface specifications of an FMU as input, and prompts an LLM to generate structured scenario goals in Given-When-Then format that define the initial input conditions of the simulation, a possible change in those conditions, and the expected output behaviour of the system against those changes. The corresponding scenario plans specify input patterns and add assertion oracles that describe expected output patterns defined in scenario goals. The approach generates a complete input time series for the scenario plans, runs the FMU simulation, and evaluates assertions on the recorded outputs. It produces human-readable logs and plots that show statistics for each scenario with overlays, aggregate pass rates, and per-goal outcomes. The generated scenarios and results are stored for evaluation and later re-execution. We evaluate the approach on a Lube Oil Cooling system and discuss design choices that make the approach practical for everyday use. Results suggest that LLM-assisted scenario generation can facilitate automatic test design and verification of dynamic simulation models.

Related papers

• Generative Artificial Intelligence (GenAI) to convert images of queuing networks into verifiable simulation models: an open-weight LLM workflow approach [0.0]
  Sketch2DES is a sketch-to-simulation workflow that converts diagrammatic representations of queuing networks into verifiable simulation models.<n>We evaluate the approach on eight queuing-network diagrams of varying complexity.
  arXiv  Detail & Related papers  (2026-07-27T10:50:37Z)
• Program Synthesis for Simulation-Based Inference: Joint Model Selection and Parameter Estimation [0.038073142980733]
  We present a framework for joint model selection and parameter estimation that combines large language models for program synthesis with neural simulation-based inference.<n>The approach enables simulation-based inference over a pool of models, not just parameters within a fixed model.
  arXiv  Detail & Related papers  (2026-07-20T04:33:35Z)
• TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulation [61.35569005726248]
  Existing embodied control research demonstrates remarkable performance improvements by scaling training data and model size.<n>Non-deterministic generative models, such as diffusion and autoregressive models, have been widely adopted in the field of embodied control.<n>We propose textbfTapSampling, a plug-and-play framework for inference-time sampling.
  arXiv  Detail & Related papers  (2026-05-25T08:03:31Z)
• Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations [1.58310730488265]
  In industrial domains, Functional Mock-up Interface (FMI) is used to exchange simulation models as Functional Mock-up Units (FMUs)<n> deriving effective test oracles for these simulation models remains challenging due to the absence of explicit expected outputs.<n>We propose an LLM-powered multi-agent workflow for specification-based metamorphic testing of FMU-based simulation models.
  arXiv  Detail & Related papers  (2026-05-24T14:30:56Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• Declarative Scenario-based Testing with RoadLogic [2.2107822884016404]
  We present RoadLogic that bridges declarative OS2 specifications and executable simulations.<n>Results show that RoadLogic consistently produces realistic, specification-satisfying simulations within minutes.
  arXiv  Detail & Related papers  (2026-03-10T10:11:09Z)
• Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving [54.46325690390831]
  We propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment.<n>MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine.<n>MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes.
  arXiv  Detail & Related papers  (2025-11-26T17:01:41Z)
• AI for Distributed Systems Design: Scalable Cloud Optimization Through Repeated LLMs Sampling And Simulators [3.1594665317979698]
  We explore AI-driven distributed-systems policy design by combining code generation from large language models with deterministic verification in a domain-specific simulator.<n>We report preliminary results on throughput improvements across multiple models.<n>We conjecture that AI will be crucial for scaling this methodology by helping to bootstrap new simulators.
  arXiv  Detail & Related papers  (2025-10-20T16:10:24Z)
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• Learning to Simulate: Generative Metamodeling via Quantile Regression [2.0613075946076904]
  Traditional metamodeling techniques learn relationships between simulator inputs and a single output summary statistic.<n>We propose a new concept: generative metamodeling.<n>Generative metamodels enable rapid generation of numerous random outputs upon input specification.
  arXiv  Detail & Related papers  (2023-11-29T16:46:24Z)
• Likelihood-Free Inference in State-Space Models with Unknown Dynamics [71.94716503075645]
  We introduce a method for inferring and predicting latent states in state-space models where observations can only be simulated, and transition dynamics are unknown. We propose a way of doing likelihood-free inference (LFI) of states and state prediction with a limited number of simulations.
  arXiv  Detail & Related papers  (2021-11-02T12:33:42Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.