STMutants: A Mutation Testing Dataset for Structured Text Programs in Industrial Automation

• URL: http://arxiv.org/abs/2606.05499v1
• Date: Wed, 03 Jun 2026 22:47:54 GMT
• Title: STMutants: A Mutation Testing Dataset for Structured Text Programs in Industrial Automation
• Abstract summary: Mutation testing is widely used to evaluate test-suite effectiveness, yet IEC 61131-3 Structured Text (ST) programs still lack a publicly available benchmark.<n>We present STMutants, a curated mutation testing dataset for industrial automation software.<n>By providing the first publicly available mutation benchmark for ST programs, STMutants enables reproducible research, mutation analysis, fault localization, and AI-assisted quality assurance for PLC software.
• Score: 0.28675177318965045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mutation testing is widely used to evaluate test-suite effectiveness, yet IEC 61131-3 Structured Text (ST) programs still lack a publicly available benchmark that supports reproducible mutation-based research. This gap is especially important because ST is extensively used in Programmable Logic Controllers (PLCs) that operate in real-time, safety-critical industrial environments, where software faults may cause equipment damage, production loss, or unsafe system behavior. To address this need, we present STMutants, a curated mutation testing dataset for industrial automation software. STMutants contains 110 generated first-order mutants derived from 11 ST programs collected from the OSCAT basic library and industrially relevant sources, of which 108 are retained after observability and equivalence screening. The dataset covers seven mutation operator categories adapted from classical taxonomies for the PLC domain, including value, relational, arithmetic, logical, negation, operation insertion/omission, and initialization faults. Each mutant is constructed through a four-phase methodology: fault-type profiling and operator selection, syntactic transformation, compilability verification, and manual equivalence screening with strong inter-rater agreement (kappa = 0.87). To demonstrate the usefulness of the dataset, we evaluate three large language models (LLMs) in a two-phase setting: test-suite generation followed by mutation kill/survive prediction. Across 108 retained mutants, the models achieve mutation detection accuracies of 86.1%, 94.4%, and 86.1%, respectively, with statistical analysis confirming significant performance differences. By providing the first publicly available mutation benchmark for ST programs, STMutants enables reproducible research on automated test generation, mutation analysis, fault localization, and AI-assisted quality assurance for PLC software.

Related papers

• Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data [73.19814469893088]
  Digital twins and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems.<n>This paper addresses whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible.<n>Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing.
  arXiv  Detail & Related papers  (2026-08-28T07:05:38Z)
• Evaluating the Effectiveness of LLMs in Aiding Compliance Testing of PKCS#1-v1.5 [0.0]
  Testing implementations of binary protocols for specification compliance requires inputs that satisfy both structural and semantic constraints.<n>This work investigates whether grammar-level mutation combined with LLM-based code synthesis can serve as a viable, more generalizable approach to specification compliance testing.
  arXiv  Detail & Related papers  (2026-06-16T18:54:57Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• VIBEPASS: Can Vibe Coders Really Pass the Vibe Check? [46.85901599242161]
  We evaluate two coupled tasks: emphFault-Triggering Test Generation (FT-Test) and emphFault-targeted Program Repair (FPR)<n>We find that fault-targeted reasoning does not scale with general coding ability.<n>Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines.
  arXiv  Detail & Related papers  (2026-03-16T21:14:28Z)
• Robust Machine Learning for Regulatory Sequence Modeling under Biological and Technical Distribution Shifts [0.3948325938742681]
  We introduce a robustness framework to quantify performance degradation, calibration failures, and uncertainty based reliability.<n>In simulation, motif driven regulatory outputs are generated with cell type specific programs, perturbations, GC bias, depth variation, batch effects, and heteroscedastic noise.<n>Models remain accurate but show higher error, severe variance miscalibration, and coverage collapse under motif effect rewiring and noise dominated regimes.
  arXiv  Detail & Related papers  (2026-01-21T13:15:27Z)
• Characterizing Bugs and Quality Attributes in Quantum Software: A Large-Scale Empirical Study [0.6445605125467574]
  This study presents the first ecosystem-scale longitudinal analysis of software bugs across 123 open source quantum repositories from 2012 to 2024.<n>Full-stack libraries and compilers are the most bug-prone categories due to circuit, gate, and transpilation-related issues.<n>High-severity bugs cluster in cryptography, experimental computing, and compiler toolchains.
  arXiv  Detail & Related papers  (2025-12-31T06:05:49Z)
• MutDafny: A Mutation-Based Approach to Assess Dafny Specifications [46.0054630008219]
  We present MutDafny, a tool that increases the reliability of Dafny specifications by automatically signaling potential weaknesses.<n>We analyze mutation operators from popular tools, identifying the ones applicable to Dafny.<n>We evaluate MutDafny's effectiveness and efficiency in a dataset of 794 real-world Dafny programs.
  arXiv  Detail & Related papers  (2025-11-19T12:58:06Z)
• User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection [0.005755004576310333]
  Insider threat detection presents unique challenges due to the authorized status of malicious actors.<n>Existing machine learning methods treat user activity as isolated events, thereby failing to leverage sequential dependencies in user behavior.<n>We propose a User-Based Sequencing (UBS) methodology, transforming the CERT insider threat dataset into structured temporal sequences suitable for deep sequential modeling.
  arXiv  Detail & Related papers  (2025-06-30T00:47:31Z)
• Latent Space Class Dispersion: Effective Test Data Quality Assessment for DNNs [45.129846925131055]
  Latent Space Class Dispersion (LSCD) is a novel metric to quantify the quality of test datasets for Deep Neural Networks (DNNs)<n>Our empirical study shows that LSCD reveals and quantifies deficiencies in the test dataset of three popular benchmarks pertaining to image classification tasks.
  arXiv  Detail & Related papers  (2025-03-24T15:45:50Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models [0.8192907805418583]
  Generative Pre-Trained Transformer models have been shown to be surprisingly effective at a variety of natural language processing tasks. We evaluate the effectiveness of open source GPT models for the task of automatic identification of the presence of vulnerable code syntax.
  arXiv  Detail & Related papers  (2024-07-31T23:33:26Z)
• A Comprehensive Study on Large Language Models for Mutation Testing [36.00296047226433]
  Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice.<n>We evaluate BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches) on 851 real bugs from two Java real-world bug benchmarks.<n>Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection.
  arXiv  Detail & Related papers  (2024-06-14T08:49:41Z)
• Contextual Predictive Mutation Testing [17.832774161583036]
  We introduce MutationBERT, an approach for predictive mutation testing that simultaneously encodes the source method mutation and test method. Thanks to its higher precision, MutationBERT saves 33% of the time spent by a prior approach on checking/verifying live mutants. We validate our input representation, and aggregation approaches for lifting predictions from the test matrix level to the test suite level, finding similar improvements in performance.
  arXiv  Detail & Related papers  (2023-09-05T17:00:15Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.