From Metrics to Improvement: A Lifecycle-Aware LLM Feedback Framework for Research Software Quality

• URL: http://arxiv.org/abs/2608.23118v1
• Date: Mon, 24 Aug 2026 11:28:26 GMT
• Title: From Metrics to Improvement: A Lifecycle-Aware LLM Feedback Framework for Research Software Quality
• Abstract summary: Research software is often developed by researchers with limited software engineering expertise.<n>This can lead to quality issues that hinder maintainability, reuse, and sustainability.<n>We propose a lifecycle-aware framework that integrates quantitative software quality assessment with Large Language Model (LLM)-based code refinement.
• Score: 8.460186703921934
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Research software is increasingly central to scientific workflows, yet it is often developed by researchers with limited software engineering expertise. This can lead to quality issues that hinder maintainability, reproducibility, reuse, and sustainability. Existing static analysis tools can identify such issues, but their outputs often require expert interpretation and provide limited support for translating quality assessments into actionable improvements. To address this gap, we propose a lifecycle-aware framework that integrates quantitative software quality assessment with Large Language Model (LLM)-based code refinement. The framework comprises two stages. First, a lifecycle-aware Quality Model is developed from established software quality standards and practitioner requirements. The model defines five quality dimensions and 25 candidate metrics, of which 14 are operationalized using existing analysis tools and custom measurements. Second, the resulting quality diagnostics are used as structured feedback within an iterative LLM-based refinement process, enabling generated improvements to be repeatedly reassessed against the Quality Model. We evaluate the framework on notebook-centric research software using multiple LLMs and compare iterative structured feedback with single-step feedback and unstructured prompting. The results show improvements in specific quality attributes, particularly code duplication and structural quality, while also revealing trade-offs among maintainability, code size, documentation, and complexity. These findings demonstrate the potential of metric-driven LLM feedback for research software quality improvement while highlighting its inherently multi-objective nature \footnote{The source code and experimental data are publicly available at https://github.com/QCDIS/Software_Quality_Control_LLM . }

Related papers

• Continuous Autonomous Refactoring: A Research Roadmap for AI-Driven Code Quality Maintenance [3.0540716731676625]
  We envision LLM-based pipelines as a continuous component of software maintenance rather than a tool invoked only for occasional manual tasks.<n>Under this vision, AI agents continuously monitor, evaluate, and improves against explicit and evolving notions of software quality.
  arXiv  Detail & Related papers  (2026-09-01T13:35:29Z)
• CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants [2.2811622267552014]
  Large Language Models (LLM) are increasingly used for software development.<n>Existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics and High Performance Computing software.<n>This paper develops practical, repeatable benchmarks that quantify LLM performance on HEP/ HPC-relevant tasks.
  arXiv  Detail & Related papers  (2026-03-01T11:16:50Z)
• Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content [71.46991494014382]
  We introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images.<n>Q-Real consists of 3,088 images generated by popular text-to-image models.<n>We construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning.
  arXiv  Detail & Related papers  (2025-11-21T02:43:17Z)
• Quality Assurance of LLM-generated Code: Addressing Non-Functional Quality Characteristics [3.0540716731676625]
  Existing studies focus mainly on whether generated code passes the tests rather than whether it passes with quality.<n>This study conducted three complementary investigations: a systematic review of 108 papers, two industry workshops with practitioners from multiple organizations, and an empirical analysis of patching real-world software issues.<n>We found that security and performance efficiency dominate academic attention, while maintainability and other qualities are understudied.
  arXiv  Detail & Related papers  (2025-11-13T12:56:07Z)
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
  evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
  arXiv  Detail & Related papers  (2025-11-07T18:01:32Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Automated Validation of LLM-based Evaluators for Software Engineering Artifacts [0.7548538278943616]
  REFINE (Ranking Evaluators for FIne grained Nuanced Evaluation) is an automated framework for benchmarking large language models (LLMs)<n> REFINE applies novel generation techniques to automatically synthesize artifacts with progressively reduced quality.<n>It quantifies each candidate evaluator configuration by measuring how closely its rankings align with expected ordering.
  arXiv  Detail & Related papers  (2025-08-04T18:52:01Z)
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval [44.39462689088926]
  CoQuIR is the first large-scale, multilingual benchmark designed to evaluate quality-aware code retrieval.<n>CoQuIR provides fine-grained quality annotations for 42,725 queries and 134,907 code snippets in 11 programming languages.
  arXiv  Detail & Related papers  (2025-05-31T13:00:17Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing [139.77117915309023]
  CRITIC allows large language models to validate and amend their own outputs in a manner similar to human interaction with tools. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs.
  arXiv  Detail & Related papers  (2023-05-19T15:19:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.