Explainability as a Compliance Requirement: What Regulated Industries Need from AI Tools for Design Artifact Generation

• URL: http://arxiv.org/abs/2507.09220v1
• Date: Sat, 12 Jul 2025 09:34:39 GMT
• Title: Explainability as a Compliance Requirement: What Regulated Industries Need from AI Tools for Design Artifact Generation
• Authors: Syed Tauhid Ullah Shah, Mohammad Hussein, Ann Barcomb, Mohammad Moshirpour,
• Abstract summary: We investigate the explainability gap in AI-driven design artifact generation through semistructured interviews with ten practitioners from safety-critical industries.<n>Our findings reveal that non-explainable AI outputs necessitate extensive manual validation, reduce stakeholder trust, struggle to handle domain-specific terminology, disrupt team collaboration, and introduce regulatory compliance risks.<n>This study outlines a practical roadmap for improving the transparency, reliability, and applicability of AI tools in requirements engineering.
• Score: 0.7874708385247352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial Intelligence (AI) tools for automating design artifact generation are increasingly used in Requirements Engineering (RE) to transform textual requirements into structured diagrams and models. While these AI tools, particularly those based on Natural Language Processing (NLP), promise to improve efficiency, their adoption remains limited in regulated industries where transparency and traceability are essential. In this paper, we investigate the explainability gap in AI-driven design artifact generation through semi-structured interviews with ten practitioners from safety-critical industries. We examine how current AI-based tools are integrated into workflows and the challenges arising from their lack of explainability. We also explore mitigation strategies, their impact on project outcomes, and features needed to improve usability. Our findings reveal that non-explainable AI outputs necessitate extensive manual validation, reduce stakeholder trust, struggle to handle domain-specific terminology, disrupt team collaboration, and introduce regulatory compliance risks, often negating the anticipated efficiency benefits. To address these issues, we identify key improvements, including source tracing, providing clear justifications for tool-generated decisions, supporting domain-specific adaptation, and enabling compliance validation. This study outlines a practical roadmap for improving the transparency, reliability, and applicability of AI tools in requirements engineering workflows, particularly in regulated and safety-critical environments where explainability is crucial for adoption and certification.

Related papers

• Rethinking Data Protection in the (Generative) Artificial Intelligence Era [115.71019708491386]
  We propose a four-level taxonomy that captures the diverse protection needs arising in modern (generative) AI models and systems.<n>Our framework offers a structured understanding of the trade-offs between data utility and control, spanning the entire AI pipeline.
  arXiv  Detail & Related papers  (2025-07-03T02:45:51Z)
• Hybrid Reasoning for Perception, Explanation, and Autonomous Action in Manufacturing [0.0]
  CIPHER is a vision-language-action (VLA) model framework aiming to replicate human-like reasoning for industrial control.<n>It integrates a process expert, a regression model enabling quantitative characterization of system states.<n>It interprets visual or textual inputs from process monitoring, explains its decisions, and autonomously generates precise machine instructions.
  arXiv  Detail & Related papers  (2025-06-10T05:37:33Z)
• Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey [59.52058740470727]
  Edge-cloud collaborative computing (ECCC) has emerged as a pivotal paradigm for addressing the computational demands of modern intelligent applications.<n>Recent advancements in AI, particularly deep learning and large language models (LLMs), have dramatically enhanced the capabilities of these distributed systems.<n>This survey provides a structured tutorial on fundamental architectures, enabling technologies, and emerging applications.
  arXiv  Detail & Related papers  (2025-05-03T13:55:38Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• Exploring Gen-AI applications in building research and industry: A review [10.154329382433213]
  This paper investigates the transformative potential of Generative AI (Gen-AI) technologies within the building industry.<n>By leveraging these advanced AI tools, the study explores their application across key areas such as automated compliance checking and building design assistance.<n>The paper concludes with a comprehensive analysis of the current capabilities of Gen-AI in the building industry.
  arXiv  Detail & Related papers  (2024-10-01T21:59:08Z)
• Retrieval-Augmented Instruction Tuning for Automated Process Engineering Calculations : A Tool-Chaining Problem-Solving Framework with Attributable Reflection [0.0]
  We introduce a novel autonomous agent framework leveraging Retrieval-Augmented Instruction-Tuning (RAIT) to enhance open, customizable small code language models (SLMs) By combining instruction tuned code SLMs with Retrieval-Augmented Code Generation (RACG) using external tools, the agent generates, debugs, and optimize code from natural language specifications. Our approach addresses the limitations of the current lack of a foundational AI model for specialized process engineering tasks and offers benefits of explainability, knowledge editing, and cost-effectiveness.
  arXiv  Detail & Related papers  (2024-08-28T15:33:47Z)
• Natural Language Processing for Requirements Traceability [47.93107382627423]
  Traceability plays a crucial role in requirements and software engineering, particularly for safety-critical systems. Natural language processing (NLP) and related techniques have made considerable progress in the past decade.
  arXiv  Detail & Related papers  (2024-05-17T15:17:00Z)
• Decoding the Digital Fine Print: Navigating the potholes in Terms of service/ use of GenAI tools against the emerging need for Transparent and Trustworthy Tech Futures [0.0]
  The research investigates the crucial role of clear and intelligible terms of service in cultivating user trust and facilitating informed decision-making in the context of AI, in specific GenAI. It highlights the obstacles presented by complex legal terminology and detailed fine print, which impede genuine user consent and recourse. Findings indicate inconsistencies and variability in document quality, signaling a pressing demand for uniformity in disclosure practices.
  arXiv  Detail & Related papers  (2024-03-26T04:54:53Z)
• AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges [60.56413461109281]
  Artificial Intelligence for IT operations (AIOps) aims to combine the power of AI with the big data generated by IT Operations processes. We discuss in depth the key types of data emitted by IT Operations activities, the scale and challenges in analyzing them, and where they can be helpful. We categorize the key AIOps tasks as - incident detection, failure prediction, root cause analysis and automated actions.
  arXiv  Detail & Related papers  (2023-04-10T15:38:12Z)
• Artificial Intelligence in Governance, Risk and Compliance: Results of a study on potentials for the application of artificial intelligence (AI) in governance, risk and compliance (GRC) [0.0]
  GRC (Governance, Risk and Compliance) means an integrated governance-approach. Governance functions are interlinked and not separated from each other. Artificial intelligence is being used in GRC for processing and analysis of unstructured data sets.
  arXiv  Detail & Related papers  (2022-12-07T12:36:10Z)
• Tools and Practices for Responsible AI Engineering [0.5249805590164901]
  We present two new software libraries that address critical needs for responsible AI engineering. hydra-zen dramatically simplifies the process of making complex AI applications, and their behaviors reproducible. The rAI-toolbox is designed to enable methods for evaluating and enhancing the robustness of AI-models.
  arXiv  Detail & Related papers  (2022-01-14T19:47:46Z)
• Counterfactual Explanations as Interventions in Latent Space [62.997667081978825]
  Counterfactual explanations aim to provide to end users a set of features that need to be changed in order to achieve a desired outcome. Current approaches rarely take into account the feasibility of actions needed to achieve the proposed explanations. We present Counterfactual Explanations as Interventions in Latent Space (CEILS), a methodology to generate counterfactual explanations.
  arXiv  Detail & Related papers  (2021-06-14T20:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.