A Benchmark Dataset And LLMs Comparison For NFR Classification With Explainable AI

• URL: http://arxiv.org/abs/2510.18096v1
• Date: Mon, 20 Oct 2025 20:45:15 GMT
• Title: A Benchmark Dataset And LLMs Comparison For NFR Classification With Explainable AI
• Abstract summary: Non-Functional Requirements (NFRs) play a critical role in determining the overall quality and user satisfaction of software systems.<n>We collected NFRs from various Project Charters and Open Source Software Documentation.<n>We categorized NFRs into sub-classes and identified needs using widely used Large Language Models.
• Score: 0.21748200848556345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Non-Functional Requirements (NFRs) play a critical role in determining the overall quality and user satisfaction of software systems. Accurately identifying and classifying NFRs is essential to ensure that software meets performance, usability, and reliability expectations. However, manual identification of NFRs from documentation is time-consuming and prone to errors, necessitating automated solutions. Before implementing any automated solution, a robust and comprehensive dataset is essential. To build such a dataset, we collected NFRs from various Project Charters and Open Source Software Documentation. This enhanced the technical depth and usability of an already existing NFR dataset. We categorized NFRs into sub-classes and identified needs using widely used Large Language Models to facilitate automation. After classifying the NFRs, we compared the classification results of the selected LLMs: RoBERTa, CodeBERT, Gemma-2, Phi-3, Mistral-8B, and Llama-3.1-8B using various evaluation metrics, including precision, recall, F1-score, and lime scores. Among these models, Gemma-2 achieved the best results with a precision of 0.87, recall of 0.89, and F1-score of 0.88, alongside a lime hit score of 78 out of 80. Phi-3 closely followed with a precision of 0.85, recall of 0.87, F1-score of 0.86, and the highest lime hit score of 79. By improving the contextual foundation, this integration enhanced the model's comprehension of technical aspects and user requirements.

Related papers

• CaresAI at SMM4H-HeaRD 2026: Predicting TNM Staging [2.7998963147546143]
  This study aims to predict Tumor, Node, and Metastasis (TNM) stage labels independently, with the Cancer Genome Atlas (TCGA) pathology report as the sixth shared task of SMM4H-HeaRD 2026.<n>We explore both classical and deep learning approaches using Term Frequency-Inverse Document Frequency (TF-IDF) features and embeddings from ClinicalBERT, BioBERT, and PubMedBERT.<n>Our results show that individual embeddings perform similarly to the TNM label classification, while their combination improves its predictive ability.
  arXiv  Detail & Related papers  (2026-07-03T16:25:35Z)
• OpenOneRec Technical Report [99.17075873619352]
  OneRec series has successfully unified the fragmented recommendation pipeline into an end-to-end generative framework.<n>OneRec Foundation (1.7B and 8B), a family of models establishing new state-of-the-art (SOTA) results across all tasks in RecIF-Bench.<n>When transferred to the Amazon benchmark, our models surpass the strongest baselines with an average 26.8% improvement in Recall@10 across 10 diverse datasets.
  arXiv  Detail & Related papers  (2025-12-31T10:15:53Z)
• Advanced Data Collection Techniques in Cloud Security: A Multi-Modal Deep Learning Autoencoder Approach [0.0]
  This research presents an innovative method to cloud security by integrating numerous data sources and modalities with multi-modal deep learning autoencoders.<n>The proposed design integrates the best features of six deep learning models: Multi-Modal Deep Learning Autoencoder (MMDLA), Anomaly Detection using Adaptive Metric Learning (ADAM), ADADELTA, ADAGRAD, RMSPROP, and Stacked Graph Transformer (SGT)
  arXiv  Detail & Related papers  (2025-11-26T17:10:54Z)
• Synergistic Feature Fusion for Latent Lyrical Classification: A Gated Deep Learning Architecture [0.0]
  This study addresses the challenge of integrating complex, high-dimensional deep semantic features with simple, interpretable structural cues for lyrical content classification.<n>We introduce a novel Synergistic Fusion Layer (SFL) architecture, a deep learning model utilizing a gated mechanism to modulate Sentence-BERT embeddings (Fdeep) using low-dimensional auxiliary features (Fstruct)<n>The SFL model achieved an accuracy of 0.9894 and a Macro F1 score of 0.9894, outperforming a comprehensive Random Forest (RF) baseline that used feature concatenation.
  arXiv  Detail & Related papers  (2025-11-11T21:12:52Z)
• Improving Crash Data Quality with Large Language Models: Evidence from Secondary Crash Narratives in Kentucky [13.50189012080061]
  This study evaluates advanced natural language processing (NLP) techniques to enhance crash data quality by mining crash narratives.<n>Drawing from 16,656 manually reviewed narratives from 2015-2022, with 3,803 confirmed secondary crashes, we compare three model classes.<n>Fine-tuned transformers achieved superior performance, with RoBERTa yielding the highest F1-score (0.90) and accuracy 95%.
  arXiv  Detail & Related papers  (2025-08-06T12:41:18Z)
• Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications [0.7124971549479361]
  This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification.<n>We determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability.
  arXiv  Detail & Related papers  (2025-05-20T21:12:58Z)
• RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation [52.87427601131587]
  We propose RobuNFR for evaluating the robustness of LLMs in NFR-aware code generation.<n>Our experiments show that RobuNFR reveals issues in the tested LLMs when considering NFRs in code generation.
  arXiv  Detail & Related papers  (2025-03-28T20:05:33Z)
• Automated Non-Functional Requirements Generation in Software Engineering with Large Language Models: A Comparative Study [0.0]
  Non-functional requirements (NFRs) are often overlooked or difficult to identify, impacting software quality.<n>We developed a framework that leverages Large Language Models (LLMs) to derive quality-driven NFRs from functional requirements (FRs)<n>Using a custom prompting technique within a Deno-based pipeline, the system identifies relevant quality attributes for each functional requirement and generates corresponding NFRs.
  arXiv  Detail & Related papers  (2025-03-19T14:23:22Z)
• SFR-RAG: Towards Contextually Faithful LLMs [57.666165819196486]
  Retrieval Augmented Generation (RAG) is a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance. We introduce SFR-RAG, a small LLM that is instruction-textual with an emphasis on context-grounded generation and hallucination. We also present ConBench, a new evaluation framework compiling multiple popular and diverse RAG benchmarks.
  arXiv  Detail & Related papers  (2024-09-16T01:08:18Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress.<n>LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset.<n>Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• Preference Learning Algorithms Do Not Learn Preference Rankings [62.335733662381884]
  We study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs. We find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets.
  arXiv  Detail & Related papers  (2024-05-29T21:29:44Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition [6.917786124918387]
  We compare Transformer models that are trained from scratch to fine-tuned BERT-based LLMs. We examine the impact of an additional CRF layer on such models to encourage contextual learning.
  arXiv  Detail & Related papers  (2022-10-23T16:27:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.