AI-Assisted Requirements Engineering: An Empirical Evaluation Relative to Expert Judgment

• URL: http://arxiv.org/abs/2604.15222v1
• Date: Thu, 16 Apr 2026 16:53:57 GMT
• Title: AI-Assisted Requirements Engineering: An Empirical Evaluation Relative to Expert Judgment
• Abstract summary: This paper investigates the extent to which AI-based tools can support systems engineers in evaluating requirement quality.<n>A controlled study was conducted in which system requirements were evaluated against established INCOSE good requirement'' criteria.<n>Results indicate that AI tools can provide consistent and rapid preliminary assessments, particularly for syntactic and structural quality attributes.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial Intelligence is increasingly introduced into systems engineering activities, particularly within requirements engineering, where quality assessment and validation remain heavily dependent on expert judgment. While recent AI tools demonstrate promising capabilities in analyzing and generating requirements, their role within formal systems engineering processes-and their alignment with established INCOSE criteria-remains insufficiently understood. This paper investigates the extent to which AI-based tools can support systems engineers in evaluating requirement quality, without replacing professional expertise. The research adopts a structured systems engineering methodology to compare AI-assisted requirement evaluation with human expert assessment. A controlled study was conducted in which system requirements were evaluated against established INCOSE ``good requirement'' criteria by both experienced systems engineers and an AI-based assessment tool. The evaluation focused on consistency, completeness, clarity, and testability, examining not only accuracy but also the decision logic underlying each assessment. Results indicate that AI tools can provide consistent and rapid preliminary assessments, particularly for syntactic and structural quality attributes. However, expert judgment remains essential for contextual interpretation, ambiguity resolution, and trade-off reasoning. Rather than positioning AI as a replacement for systems engineers, the findings support its role as a decision-support mechanism within the RE lifecycle. From a systems engineering perspective, this study contributes empirical evidence on how AI can be integrated into RE workflows while preserving traceability, accountability, and engineering consistency.

Related papers

• AI Deployment Accountability Engineering: A Vision for Accountable AI in Safety-Critical Socio-Technical Systems [14.280538199149634]
  This vision paper introduces AI Deployment Accountability Engineering (ADAE)<n>ADAE is a proposed AI engineering subdiscipline concerned with establishing measurable, continuous, and actionable accountability for deployed AI systems.<n>We articulate a research agenda built around four interconnected pillars: structured discovery of context-dependent failure modes, privacy-preserving accountability measurement, system-level risk analysis for agentic AI, and translation of technical failures into operational, and institutional risks.
  arXiv  Detail & Related papers  (2026-09-13T15:18:41Z)
• Integrating AI into Requirements Quality Learning in Software Engineering Education: A TPACK-Guided Empirical Study [0.43398392878696535]
  Students used the AI tool selectively, mainly as support for analysis and evaluation.<n>Students reported conditional trust, active refinement, and increased awareness of quality criteria.
  arXiv  Detail & Related papers  (2026-07-30T13:15:16Z)
• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• Towards the Next Generation of Software: Insights from Grey Literature on AI-Native Applications [13.876049229274114]
  AI-native applications are a new paradigm in software engineering that fundamentally redefines how software is designed, developed, and evolved.<n>Despite their growing prominence, AI-native applications still lack a unified engineering definition and architectural blueprint.<n>This study seeks to establish a comprehensive understanding of AI-native applications by identifying their defining characteristics, key quality attributes, and typical technology stacks.
  arXiv  Detail & Related papers  (2025-09-16T15:01:23Z)
• Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned [45.44933002008943]
  This white paper presents the T"UV AUSTRIA Trusted AI framework.<n>It is an end-to-end audit catalog and methodology for assessing and certifying machine learning systems.<n>Building on three pillars - Secure Software Development, Functional Requirements, and Ethics & Data Privacy - it translates the high-level obligations of the EU AI Act into specific, testable criteria.
  arXiv  Detail & Related papers  (2025-09-08T17:52:08Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• On the Evaluation of Engineering Artificial General Intelligence [5.802869598386355]
  We discuss the challenges and propose a framework for evaluating engineering artificial general intelligence (eAGI) agents.<n>We consider eAGI as a specialization of artificial general intelligence (AGI)<n>eAGI agents should possess a unique blend of background knowledge (recall and retrieve) of facts and methods.
  arXiv  Detail & Related papers  (2025-05-15T18:52:47Z)
• Work in Progress: AI-Powered Engineering-Bridging Theory and Practice [0.0]
  This paper explores how generative AI can help automate and improve key steps in systems engineering.<n>It examines AI's ability to analyze system requirements based on INCOSE's "good requirement" criteria.<n>The research aims to assess AI's potential to streamline engineering processes and improve learning outcomes.
  arXiv  Detail & Related papers  (2025-02-06T17:42:00Z)
• A Unified Framework for Evaluating the Effectiveness and Enhancing the Transparency of Explainable AI Methods in Real-World Applications [2.0681376988193843]
  This study introduces a single evaluation framework for XAI.<n>It uses both numbers and user feedback to check if the explanations are correct, easy to understand, fair, complete, and reliable.<n>We show the value of this framework through case studies in healthcare, finance, farming, and self-driving systems.
  arXiv  Detail & Related papers  (2024-12-05T05:30:10Z)
• Design of a Quality Management System based on the EU Artificial Intelligence Act [0.0]
  The EU AI Act mandates that providers and deployers of high-risk AI systems establish a quality management system (QMS) This paper introduces a new design concept and prototype for a QMS as a microservice Software as a Service web application.
  arXiv  Detail & Related papers  (2024-08-08T12:14:02Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)
• Evaluating General-Purpose AI with Psychometrics [43.85432514910491]
  We discuss the need for a comprehensive and accurate evaluation of general-purpose AI systems such as large language models. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation.
  arXiv  Detail & Related papers  (2023-10-25T05:38:38Z)
• Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
  This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
  arXiv  Detail & Related papers  (2022-09-07T10:09:12Z)
• An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
  This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
  arXiv  Detail & Related papers  (2021-05-07T12:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.