Automated Non-Functional Requirements Generation in Software Engineering with Large Language Models: A Comparative Study

• URL: http://arxiv.org/abs/2503.15248v1
• Date: Wed, 19 Mar 2025 14:23:22 GMT
• Title: Automated Non-Functional Requirements Generation in Software Engineering with Large Language Models: A Comparative Study
• Abstract summary: Non-functional requirements (NFRs) are often overlooked or difficult to identify, impacting software quality.<n>We developed a framework that leverages Large Language Models (LLMs) to derive quality-driven NFRs from functional requirements (FRs)<n>Using a custom prompting technique within a Deno-based pipeline, the system identifies relevant quality attributes for each functional requirement and generates corresponding NFRs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neglecting non-functional requirements (NFRs) early in software development can lead to critical challenges. Despite their importance, NFRs are often overlooked or difficult to identify, impacting software quality. To support requirements engineers in eliciting NFRs, we developed a framework that leverages Large Language Models (LLMs) to derive quality-driven NFRs from functional requirements (FRs). Using a custom prompting technique within a Deno-based pipeline, the system identifies relevant quality attributes for each functional requirement and generates corresponding NFRs, aiding systematic integration. A crucial aspect is evaluating the quality and suitability of these generated requirements. Can LLMs produce high-quality NFR suggestions? Using 34 functional requirements - selected as a representative subset of 3,964 FRs-the LLMs inferred applicable attributes based on the ISO/IEC 25010:2023 standard, generating 1,593 NFRs. A horizontal evaluation covered three dimensions: NFR validity, applicability of quality attributes, and classification precision. Ten industry software quality evaluators, averaging 13 years of experience, assessed a subset for relevance and quality. The evaluation showed strong alignment between LLM-generated NFRs and expert assessments, with median validity and applicability scores of 5.0 (means: 4.63 and 4.59, respectively) on a 1-5 scale. In the classification task, 80.4% of LLM-assigned attributes matched expert choices, with 8.3% near misses and 11.3% mismatches. A comparative analysis of eight LLMs highlighted variations in performance, with gemini-1.5-pro exhibiting the highest attribute accuracy, while llama-3.3-70B achieved higher validity and applicability scores. These findings provide insights into the feasibility of using LLMs for automated NFR generation and lay the foundation for further exploration of AI-assisted requirements engineering.

Related papers

• COOPA: A Modular LLM Agent Architecture for Operations Research Problems [45.369080363530976]
  Operations Research (OR) provides a rigorous framework for high-stakes decision-making.<n>Recent LLM-based systems automate parts of this pipeline, yet remain limited by low accuracy on complex problems.<n>We propose COOPA, a modular LLM-agent architecture for interpretable and scalable OR decision support.
  arXiv  Detail & Related papers  (2026-06-25T23:56:58Z)
• Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks [45.86413490112477]
  Large language models (LLMs) are used in reliability-sensitive applications.<n>Estimateing rare failure probability with tight confidence bounds requires prohibitively large LLM inference sizes.<n>We propose to learn a sampling distribution concentrated on failure-prone inputs via the cross-entropy method.
  arXiv  Detail & Related papers  (2026-05-11T20:23:44Z)
• Domain Adaptation of Large Language Models for Polymer-Composite Additive Manufacturing Using Retrieval-Augmented Generation and Fine-Tuning [0.0]
  General-purpose large language models (LLMs) often struggle to generate reliable responses in specialized engineering domains.<n>This study investigates strategies for adapting a foundation LLM to the additive manufacturing (AM) domain.
  arXiv  Detail & Related papers  (2026-04-02T20:52:34Z)
• An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data [0.5156484100374058]
  Retrieval-augmented generation (RAG) shows promise for knowledge-intensive tasks, but no prior work has evaluated RAG on authentic industrial RE.<n>This paper presents a comprehensive empirical evaluation of RAG for industrial requirements engineering automation using authentic automotive manufacturing documentation.<n>RAG achieves 98.2% extraction accuracy with complete traceability, outperforming baselines by 24.4% and 19.6%, respectively.
  arXiv  Detail & Related papers  (2026-03-20T22:05:08Z)
• Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation [0.0]
  This paper compares five Large Language Models (LLMs) deployed between April 2024 and August 2025 for medical QA.<n>Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini.<n>Results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks.
  arXiv  Detail & Related papers  (2026-02-16T08:53:23Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• A Benchmark Dataset And LLMs Comparison For NFR Classification With Explainable AI [0.21748200848556345]
  Non-Functional Requirements (NFRs) play a critical role in determining the overall quality and user satisfaction of software systems.<n>We collected NFRs from various Project Charters and Open Source Software Documentation.<n>We categorized NFRs into sub-classes and identified needs using widely used Large Language Models.
  arXiv  Detail & Related papers  (2025-10-20T20:45:15Z)
• LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models [51.55869466207234]
  Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting.<n>We introduce LLMEval-3, a framework for dynamic evaluation of LLMs.<n>LLEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run.
  arXiv  Detail & Related papers  (2025-08-07T14:46:30Z)
• Classification of Quality Characteristics in Online User Feedback using Linguistic Analysis, Crowdsourcing and LLMs [0.9565934024763957]
  Online user feedback is a valuable source of quality-related feedback on software products.<n>The abundance of online user feedback warrants the automated identification of quality characteristics.<n>We investigate the viability of three approaches that could be effective in low-data settings.
  arXiv  Detail & Related papers  (2025-06-13T12:37:07Z)
• Structuring Radiology Reports: Challenging LLMs with Lightweight Models [5.01440254761063]
  Large language models (LLMs) have shown strong capabilities in reformatting clinical text, their high computational requirements, lack of transparency, and data privacy concerns hinder practical deployment.<n>We explore lightweight encoder-decoder models (300M parameters)-specifically T5 and BERT2BERT-for structuring radiology reports from the MIMIC-CXR and CheXpert Plus datasets.<n>Our best-performing lightweight model outperforms all LLMs adapted using prompt-based techniques on a human-annotated test set.
  arXiv  Detail & Related papers  (2025-05-30T20:12:51Z)
• ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation [4.475603469482274]
  Software engineers can engage with ReqBrain through chat-based sessions to automatically generate software requirements.<n>Top-performing model, Zephyr-7b-beta, achieved 89.30% Fl using the BERT score and a FRUGAL score of 91.20 in generating authentic and adequate requirements.
  arXiv  Detail & Related papers  (2025-05-23T08:45:46Z)
• From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering [0.7874708385247352]
  This paper explores the use of Large Language Models (LLMs) to improve qualitative data analysis tasks in Requirements Engineering (RE) We show that GPT-4 achieves substantial agreement with human analysts in deductive settings, with Cohen's Kappa scores exceeding 0.7, while zero-shot performance remains limited. These findings highlight the potential of LLMs to support QDA in RE by reducing manual effort while maintaining annotation quality.
  arXiv  Detail & Related papers  (2025-04-27T23:21:52Z)
• RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation [52.87427601131587]
  We propose RobuNFR for evaluating the robustness of LLMs in NFR-aware code generation. Our experiments show that RobuNFR reveals issues in the tested LLMs when considering NFRs in code generation.
  arXiv  Detail & Related papers  (2025-03-28T20:05:33Z)
• AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models [86.83875864328984]
  We propose an automated method for synthesizing open-ended logic puzzles, and use it to develop a bilingual benchmark, AutoLogi.<n>Our approach features program-based verification and controllable difficulty levels, enabling more reliable evaluation that better distinguishes models' reasoning abilities.
  arXiv  Detail & Related papers  (2025-02-24T07:02:31Z)
• Analysis of LLMs vs Human Experts in Requirements Engineering [0.0]
  Large Language Models (LLM) application to software development has been on the subject of code generation.<n>This study compares LLM's ability to elicit requirements of a software system, as compared to that of a human expert in a time-boxed and prompt-boxed study.
  arXiv  Detail & Related papers  (2025-01-31T16:55:17Z)
• Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation [55.21013307734612]
  AoPS-Instruct is a dataset of more than 600,000 high-quality QA pairs.<n>LiveAoPSBench is an evolving evaluation set with timestamps, derived from the latest forum data.<n>Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning.
  arXiv  Detail & Related papers  (2025-01-24T06:39:38Z)
• AttackQA: Development and Adoption of a Dataset for Assisting Cybersecurity Operations using Fine-tuned and Open-Source LLMs [0.0]
  Large language models (LLMs) are fine-tuned for generating responses to user queries. In this study, we develop a cybersecurity question-answering (Q&A) dataset, called AttackQA. We employ it to build a RAG-based Q&A system designed for analysts in security operations centers.
  arXiv  Detail & Related papers  (2024-11-01T23:03:40Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input.<n>GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• SFR-RAG: Towards Contextually Faithful LLMs [57.666165819196486]
  Retrieval Augmented Generation (RAG) is a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance. We introduce SFR-RAG, a small LLM that is instruction-textual with an emphasis on context-grounded generation and hallucination. We also present ConBench, a new evaluation framework compiling multiple popular and diverse RAG benchmarks.
  arXiv  Detail & Related papers  (2024-09-16T01:08:18Z)
• A Fuzzy Logic-Based Quality Model For Identifying Microservices With Low Maintainability [0.0]
  This paper proposes a hierarchical quality model based on fuzzy logic to measure and evaluate the maintainability of MSAs. We use a fuzzification technique to transform crisp values of code metrics into fuzzy levels and apply them as inputs to our quality model.
  arXiv  Detail & Related papers  (2024-06-20T16:53:37Z)
• Investigating Automatic Scoring and Feedback using Large Language Models [46.1232919707345]
  This paper explores the efficacy of PEFT-based quantized models, employing classification or regression head, to fine-tune language models for automatic grading and feedback generation. The results show that prediction of grade scores via finetuned LLMs are highly accurate, achieving less than 3% error in grade percentage on average.
  arXiv  Detail & Related papers  (2024-05-01T16:13:54Z)
• FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability [70.84333325049123]
  FoFo is a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats. This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats.
  arXiv  Detail & Related papers  (2024-02-28T19:23:27Z)
• Large Language Models as Automated Aligners for benchmarking Vision-Language Models [48.4367174400306]
  Vision-Language Models (VLMs) have reached a new level of sophistication, showing notable competence in executing intricate cognition and reasoning tasks. Existing evaluation benchmarks, primarily relying on rigid, hand-crafted datasets, face significant limitations in assessing the alignment of these increasingly anthropomorphic models with human intelligence. In this work, we address the limitations via Auto-Bench, which delves into exploring LLMs as proficient curation, measuring the alignment betweenVLMs and human intelligence and value through automatic data curation and assessment.
  arXiv  Detail & Related papers  (2023-11-24T16:12:05Z)
• On Systematically Building a Controlled Natural Language for Functional Requirements [2.9676973500772887]
  Natural language (NL) is pervasive in software requirements specifications (SRSs) Despite its popularity and widespread use, NL is highly prone to quality issues such as vagueness, ambiguity, and incompleteness. Controlled natural languages (CNLs) have been proposed as a way to prevent quality problems in requirements documents.
  arXiv  Detail & Related papers  (2020-05-04T09:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.