RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation

• URL: http://arxiv.org/abs/2503.22851v2
• Date: Thu, 03 Apr 2025 00:55:35 GMT
• Title: RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation
• Abstract summary: We propose RobuNFR for evaluating the robustness of LLMs in NFR-aware code generation.<n>Our experiments show that RobuNFR reveals issues in the tested LLMs when considering NFRs in code generation.
• Score: 52.87427601131587
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When using LLMs to address Non-Functional Requirements (NFRs), developers may behave differently (e.g., expressing the same NFR in different words). Robust LLMs should output consistent results across these variations; however, this aspect remains underexplored. We propose RobuNFR for evaluating the robustness of LLMs in NFR-aware code generation across four NFR dimensions: design, readability, reliability, and performance, using three methodologies: prompt variation, regression testing, and diverse workflows. Our experiments show that RobuNFR reveals robustness issues in the tested LLMs when considering NFRs in code generation. Specifically, under prompt variation, including NFRs leads to a decrease in Pass@1 by up to 39 percent and an increase in the standard deviation from 0.48 to 2.48 compared to the baseline without NFRs (i.e., Function-Only). While incorporating NFRs generally improves overall NFR metrics, it also results in higher prompt sensitivity. In regression settings, some LLMs exhibit differences across versions, with improvements in one aspect (e.g., reduced code smells) often accompanied by regressions in another (e.g., decreased correctness), revealing inconsistencies that challenge their robustness. When varying workflows, the tested LLMs show significantly different NFR-aware code generation capabilities between two workflows: (1) integrating NFRs and functional requirements into the initial prompt and (2) enhancing Function-Only-generated code with the same NFR.

Related papers

• Beyond Fixed Fault Models: Comparing LLM-Based and Rule-Based Fault Injection in OpenStack [3.6223576377024034]
  Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation.<n>Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually.<n>Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults.
  arXiv  Detail & Related papers  (2026-09-08T12:51:19Z)
• Does ISO-Grounded NFR Specification Improve LLM Code Generation? A Comparison of Rich and Structured Interventions against a Natural-Language Baseline [0.4640835690336653]
  We ask whether Non-Functional Requirements (NFRs) are rich natural prose (NL-rich) or structured grounding (Structured)<n>We evaluate four NFRs (performance, error handling, code smell, readability) with ten prompt variations per condition under a fixed model snapshot.<n>For error handling, extended-test pass rate decreases, suggesting tension between defensive coding patterns and exact-output benchmarks.
  arXiv  Detail & Related papers  (2026-08-13T19:58:01Z)
• NFR-to-Code Traceability in a Blockchain-IoT System: An Empirical Study [14.238775682979513]
  This paper studies NFR-to-Code traceability based on a real-world blockchain-IoT project.<n>We design an annotation protocol for constructing trace links across heterogeneous artifacts and build a manually curated subset containing both FR and NFR links.<n>The results show that FR-to-Code tracing is consistently easier than NFR-to-Code tracing, while security-related NFRs are the most difficult subset.
  arXiv  Detail & Related papers  (2026-07-28T06:20:33Z)
• Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization [1.5647246115099227]
  Interpretable Mesomorphic Neural Networks (IMNs) offer a promising framework that combines the predictive power of deep neural networks with the interpretability of linear models.<n>We introduce Local Fidelity Regularization (LFR), a novel penalty term that prevents degenerate weight collapse.<n>This structural constraint guarantees faithful explanations and substantially improves the reliability of model interpretations.
  arXiv  Detail & Related papers  (2026-06-29T08:26:20Z)
• From Restructuring to Stabilization: A Large-Scale Experiment on Iterative Code Readability Refactoring with Large Language Models [5.31828955342405]
  Large language models (LLMs) are increasingly used for automated code tasks.<n>This article systematically study the capabilities of LLMs for code readability.
  arXiv  Detail & Related papers  (2026-02-25T12:05:25Z)
• Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model [98.35868970993232]
  Diffusion language models (DLMs) are emerging as a powerful and promising alternative to the dominant autoregressive paradigm.<n>We introduce efficient Sampling with Adaptive acceleration and Backtracking Enhanced Remasking (i.e., Saber) to achieve better inference speed and output quality in code generation.
  arXiv  Detail & Related papers  (2025-10-20T23:38:12Z)
• Enhancing LLM-based Fault Localization with a Functionality-Aware Retrieval-Augmented Generation Framework [14.287359838639608]
  FaR-Loc is a framework that enhances method-level fault localization.<n> FaR-Loc consists of three key components: LLM Functionality Extraction, Semantic Retrieval, and LLM Re-ranking.<n>Our experiments on the widely used Defects4J benchmark show that FaR-Loc outperforms state-of-the-art LLM-based baselines.
  arXiv  Detail & Related papers  (2025-09-24T20:37:11Z)
• Semantic-Aware Fuzzing: An Empirical Framework for LLM-Guided, Reasoning-Driven Input Mutation [0.5336076422485075]
  Security vulnerabilities in Internet-of-Things devices, mobile platforms, and autonomous systems remain critical.<n>Traditional mutation-based fuzzers primarily perform byte or bit-level edits without semantic reasoning.<n>We present an open-source framework that integrates reasoning LLMs with AFL++ on Google's FuzzBench.
  arXiv  Detail & Related papers  (2025-09-23T19:57:29Z)
• Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing [13.099973383252452]
  Large language models (LLMs) are highly sensitive to prompt perturbations.<n>We introduce CREME, a novel approach that enhances LLM robustness through targeted parameter updates.<n> Experimental results show that CREME improves Pass@1 accuracy by 63% on perturbed prompts.
  arXiv  Detail & Related papers  (2025-07-22T09:57:55Z)
• Enhancing the Robustness of LLM-Generated Code: Empirical Study and Framework [25.793118619876513]
  RobGen is a framework designed to enhance code robustness without requiring model retraining.<n>RobGen reduces the proportion of less robust model-generated code by 20.0%.
  arXiv  Detail & Related papers  (2025-03-26T03:44:03Z)
• The Power of Negative Zero: Datatype Customization for Quantized Large Language Models [5.503925076208333]
  Post-training quantization serves as one of the most hardware-efficient methods to mitigate the memory and computational demands of large language models (LLMs) In this paper, we extend the basic FP datatype to perform Redundant Zero Remapping (RaZeR) RaZeR remaps the negative zero FP encoding to a set of pre-defined special values to maximally utilize FP quantization encodings and to better fit numerical distributions.
  arXiv  Detail & Related papers  (2025-01-06T22:40:40Z)
• SFR-RAG: Towards Contextually Faithful LLMs [57.666165819196486]
  Retrieval Augmented Generation (RAG) is a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance. We introduce SFR-RAG, a small LLM that is instruction-textual with an emphasis on context-grounded generation and hallucination. We also present ConBench, a new evaluation framework compiling multiple popular and diverse RAG benchmarks.
  arXiv  Detail & Related papers  (2024-09-16T01:08:18Z)
• FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models [50.331708897857574]
  We introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications. FactorLLM achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed.
  arXiv  Detail & Related papers  (2024-08-15T16:45:16Z)
• Applying RLAIF for Code Generation with API-usage in Lightweight LLMs [15.366324461797582]
  Reinforcement Learning from AI Feedback (RLAIF) has demonstrated significant potential across various domains. This paper introduces an RLAIF framework for improving the code generation abilities of lightweight (1B parameters) LLMs.
  arXiv  Detail & Related papers  (2024-06-28T17:16:03Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• LLMs for Generating and Evaluating Counterfactuals: A Comprehensive Study [2.7731115923558143]
  Large Language Models (LLMs) have shown remarkable performance in NLP tasks, but their efficacy in generating high-quality Counterfactuals (CFs) remains uncertain. We compare several common LLMs and evaluate their CFs, assessing both intrinsic metrics, and the impact of these CFs on data augmentation. Our results show that LLMs generate fluent CFs, but struggle to keep the induced changes minimal.
  arXiv  Detail & Related papers  (2024-04-26T11:57:21Z)
• FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping [49.66872823080736]
  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. To mitigate overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. We propose FFN-SkipLLM, which is an input-adaptive feed-forward skipping strategy.
  arXiv  Detail & Related papers  (2024-04-05T02:35:43Z)
• Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation [31.657608562937543]
  We introduce GRIFFIN, a training-free and calibration-free method that selects unique FF experts at the sequence level for efficient generation. GRIFFIN maintains the original model's performance with little to no degradation on a variety of classification and generation tasks.
  arXiv  Detail & Related papers  (2024-04-01T17:56:06Z)
• Assessing the Reliability of Large Language Model Knowledge [78.38870272050106]
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? We propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs' factual reliability.
  arXiv  Detail & Related papers  (2023-10-15T12:40:30Z)
• Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications [3.6048665052465667]
  We introduce RSMI, a novel two-stage framework that combines randomized smoothing (RS) with masked inference (MI) to improve the adversarial robustness of NLP systems. RS transforms a classifier into a smoothed classifier to obtain robust representations, whereas MI forces a model to exploit the surrounding context of a masked token in an input sequence. RSMI improves adversarial robustness by 2 to 3 times over existing state-of-the-art methods on benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-11T01:50:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.