Beyond Fixed Fault Models: Comparing LLM-Based and Rule-Based Fault Injection in OpenStack

• URL: http://arxiv.org/abs/2609.08681v1
• Date: Tue, 08 Sep 2026 12:51:19 GMT
• Title: Beyond Fixed Fault Models: Comparing LLM-Based and Rule-Based Fault Injection in OpenStack
• Abstract summary: Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation.<n>Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually.<n>Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults.
• Score: 3.6223576377024034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation. Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually. Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults. We compare two code LLMs, Qwen2.5-Coder and DeepSeek-Coder, with ProFIPy in OpenStack's Nova and Cinder services. On shared injection targets, activation and observable-failure rates are comparable, but operational profiles differ: LLM-generated faults produce more Catastrophic outcomes on Nova, whereas ProFIPy produces more Silent and Multi-component effects. The sampled LLM outputs also differ in how they manifest failure, while showing greater agreement in their propagation scope. These findings show that LLM-based fault injection extends the behavioral coverage of fixed fault models without establishing general superiority, and that practical adoption still requires controlled generation, runtime validation, system-level oracles, and reproducible experimental provenance.

Related papers

• LLM Detection as an Intervention: Downstream Impact under Strategic User Behavior [53.63441578662801]
  Detectors operate as an intervention that steers not only the detected attribute itself, but also downstream metrics such as LLM usage and output quality.<n>We develop a stylized model which captures how users strategically choose how much to use the LLM and how to post-process content to reduce the detected attribute.<n>We find that even when reducing the detected attribute improves output quality, introducing an LLM detector can lead users to produce lower quality outputs.
  arXiv  Detail & Related papers  (2026-07-21T17:11:02Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Exploring LLM-based Frameworks for Fault Diagnosis [2.2562573557834686]
  Large Language Model (LLM)-based systems present new opportunities for autonomous health monitoring in sensor-rich industrial environments.<n>This study explores the potential of LLMs to detect and classify faults directly from sensor data, while producing inherently explainable outputs through natural language reasoning.
  arXiv  Detail & Related papers  (2025-09-27T04:53:15Z)
• When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning [0.8880611506199766]
  We show how prompt format, model behavior, and structural assumptions influence both success rates and failure characteristics.<n>Our analysis reveals a diverse set of error-prone behaviors, including format-induced misinterpretations and runtime-disruptive code that compiles but breaks downstream.
  arXiv  Detail & Related papers  (2025-09-13T19:00:04Z)
• Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs [5.10123605644148]
  Automated Vulnerability Repair (AVR) is a fast-growing branch of program repair.<n>Recent studies show that large language models (LLMs) outperform traditional techniques.
  arXiv  Detail & Related papers  (2025-07-28T16:39:16Z)
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding [51.711605076319216]
  Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities.<n>We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.<n>We propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality.
  arXiv  Detail & Related papers  (2025-05-28T17:39:15Z)
• Teaching Your Models to Understand Code via Focal Preference Alignment [70.71693365502212]
  In existing approaches, a set of n candidate solutions is evaluated based on test case success rates.<n>Because this approach aligns entire failing code blocks rather than pinpointing specific errors, it lacks the granularity necessary to capture meaningful error-correction relationships.<n>We propose Target-DPO, a new preference alignment framework that mimics human iterative debug to refine Code LLMs.
  arXiv  Detail & Related papers  (2025-03-04T16:56:34Z)
• Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output [49.893971654861424]
  We present a light-weight approach for detecting nonfactual outputs from retrieval-augmented generation (RAG) We compute a factuality score that can be thresholded to yield a binary decision. Our experiments show high area under the ROC curve (AUC) across a wide range of relevant open source datasets.
  arXiv  Detail & Related papers  (2024-11-01T20:44:59Z)
• DALD: Improving Logits-based Detector without Logits from Black-box LLMs [56.234109491884126]
  Large Language Models (LLMs) have revolutionized text generation, producing outputs that closely mimic human writing. We present Distribution-Aligned LLMs Detection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations.
  arXiv  Detail & Related papers  (2024-06-07T19:38:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.