Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection

• URL: http://arxiv.org/abs/2608.21298v1
• Date: Fri, 21 Aug 2026 16:58:48 GMT
• Title: Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection
• Abstract summary: Large language models (LLMs) have stimulated interest in their potential to support requirements engineering tasks.<n>We aim to investigate the impact of LLM support on human-performed RI.
• Score: 41.425568895513806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background. Requirements inspection (RI) is a well-established practice for detecting potential defects in requirements artifacts early in the software lifecycle. Recent advances in large language models (LLMs) have stimulated interest in their potential to support requirements engineering (RE) tasks. However, empirical evidence on the effects of LLMs when used as collaborative assistants in human-performed RI remains scarce. Aims. We aim to investigate the impact of LLM support on human-performed RI, considering inspection effectiveness in terms of smell identification and severity classification (i.e., nocuous vs innocuous), as well as inspection duration. Method. We conducted a controlled crossover design experiment with 34 participants, who inspected textual specifications with and without LLM support, identifying and classifying requirements smells while recording inspection time. We analyzed the data using one Bayesian regression model per outcome variable, accounting for validity threats induced by the crossover design as well as covariates and mediators. Results. Results show that LLM support negatively affects smell detection accuracy but has no significant effect on smell classification or task duration. A learning effect is present across experimental periods, but reduced when RI is first performed with LLM support. Conclusions. Our findings provide empirical evidence that LLM support does not necessarily improve performance and may, instead, hinder it for novice inspectors. Moreover, the results suggest that learning RI with LLM-support from the beginning may slow down the skill acquisition process, implying threats for LLM-supported learning.

Related papers

• LLM Detection as an Intervention: Downstream Impact under Strategic User Behavior [53.63441578662801]
  Detectors operate as an intervention that steers not only the detected attribute itself, but also downstream metrics such as LLM usage and output quality.<n>We develop a stylized model which captures how users strategically choose how much to use the LLM and how to post-process content to reduce the detected attribute.<n>We find that even when reducing the detected attribute improves output quality, introducing an LLM detector can lead users to produce lower quality outputs.
  arXiv  Detail & Related papers  (2026-07-21T17:11:02Z)
• Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference [0.509780930114934]
  We study when a treatment effect estimated on large language models (LLMs) can recover the effect for the human population of interest.<n>We develop a statistical framework that adapts surrogate endpoint theory to LLMs.<n>A central takeaway is that A/B testing on LLM responses is correct only by assumption, whereas A/B testing on humans is correct by design.
  arXiv  Detail & Related papers  (2026-06-15T18:06:20Z)
• Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs [0.29737220923371155]
  Large Language Models (LLMs) are used for study screening in systematic literature reviews.<n>Despite rapid uptake, there is limited evidence on how such models behave during the study screening phase.<n>This study investigated the impact of input metadata (abstract, title, keywords) on LLM performance.
  arXiv  Detail & Related papers  (2026-04-29T08:55:16Z)
• IV Co-Scientist: Multi-Agent LLM Framework for Causal Instrumental Variable Discovery [61.15184885636171]
  In the presence of confounding between an endogenous variable and the outcome, instrumental variables (IVs) are used to isolate the causal effect of the endogenous variable.<n>We investigate whether large language models (LLMs) can aid in this task.<n>We introduce IV Co-Scientist, a multi-agent system that proposes, critiques, and refines IVs for a given treatment-outcome pair.
  arXiv  Detail & Related papers  (2026-02-08T12:28:29Z)
• CARE: Turning LLMs Into Causal Reasoning Expert [14.390784501489733]
  We show that large language models (LLMs) lack the ability to identify causal relationships.<n>We propose CARE, a framework that enhances LLMs' causal-reasoning ability.
  arXiv  Detail & Related papers  (2025-11-20T03:34:16Z)
• On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability [45.24937784556523]
  We investigate the role of requirements smells-indicators of potential issues like ambiguity and inconsistency-when used in prompts for large language models (LLMs)<n>Our results show mixed outcomes: while requirements smells had a small but significant effect when predicting whether a requirement was implemented in a piece of code (i.e., a trace link exists), no significant effect was observed when tracing the requirements with the associated lines of code.<n>These findings suggest that requirements smells can affect LLM performance in certain SE tasks but may not uniformly impact all tasks.
  arXiv  Detail & Related papers  (2025-01-08T19:54:31Z)
• Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement [51.601916604301685]
  Large language models (LLMs) generate content that can undermine trust in online discourse.<n>Current methods often focus on binary classification, failing to address the complexities of real-world scenarios like human-LLM collaboration.<n>To move beyond binary classification and address these challenges, we propose a new paradigm for detecting LLM-generated content.
  arXiv  Detail & Related papers  (2024-10-18T08:14:10Z)
• The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? [60.01746782465275]
  Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership.
  arXiv  Detail & Related papers  (2024-10-07T02:30:18Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)
• How Good Are LLMs at Out-of-Distribution Detection? [13.35571704613836]
  Out-of-distribution (OOD) detection plays a vital role in enhancing the reliability of machine learning (ML) models. This paper embarks on a pioneering empirical investigation of OOD detection in the domain of large language models (LLMs)
  arXiv  Detail & Related papers  (2023-08-20T13:15:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.