Challenges and Contributions in Quality of AI-Based Software: A Systematic Mapping Study

• URL: http://arxiv.org/abs/2608.26215v1
• Date: Wed, 26 Aug 2026 09:08:06 GMT
• Title: Challenges and Contributions in Quality of AI-Based Software: A Systematic Mapping Study
• Abstract summary: The study synthesizes primary studies published between January 2020 and January 2026 and selected from five electronic data sources.<n>The results identify six recurring challenge categories, with the most prominent being limitations in existing quality assessment models.<n>The findings suggest a call for collaboration of researchers and industrial practitioners with standardization organizations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Artificial Intelligence (AI) is increasingly embedded in modern software systems, raising important questions about how its quality should be defined, assessed, and assured. This paper presents a Systematic Mapping Study (SMS) on the quality of AI-based software. The study synthesizes primary studies published between January 2020 and January 2026 and selected from five electronic data sources. A total of 33 primary studies were included after automated search, screening, and snowballing. The results identify six recurring challenge categories, with the most prominent being limitations in existing quality assessment models, followed by issues in non-functional requirement management, quality-aware development, and quality assurance. The findings suggest a call for collaboration of researchers and industrial practitioners with standardization organizations, that could possibly devise comprehensive quality assessments and their measurement methods.

Related papers

• AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps [0.0]
  Software testing is critical for verifying that systems meet specified requirements, yet remains among the most time-consuming and expensive activities in development.<n>Recent advances in AI, natural language processing (NLP), and large language models (LLMs) have made automating this pipeline increasingly feasible.<n>This survey addresses four research questions: what AI and NLP techniques have been proposed for generating test cases from natural language requirements; what tools and frameworks support these approaches; and what research gaps remain.
  arXiv  Detail & Related papers  (2026-06-04T15:07:23Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• From Task Executors to Research Partners: Evaluating AI Co-Pilots Through Workflow Integration in Biomedical Research [0.16174969956296248]
  This rapid review examines benchmarking practices for AI systems in preclinical biomedical research.<n>A process-oriented evaluation framework is proposed that addresses four critical dimensions absent from current benchmarks.<n>These dimensions are essential for evaluating AI systems as research co-pilots rather than as isolated task executors.
  arXiv  Detail & Related papers  (2025-12-04T14:37:46Z)
• Towards the Next Generation of Software: Insights from Grey Literature on AI-Native Applications [13.876049229274114]
  AI-native applications are a new paradigm in software engineering that fundamentally redefines how software is designed, developed, and evolved.<n>Despite their growing prominence, AI-native applications still lack a unified engineering definition and architectural blueprint.<n>This study seeks to establish a comprehensive understanding of AI-native applications by identifying their defining characteristics, key quality attributes, and typical technology stacks.
  arXiv  Detail & Related papers  (2025-09-16T15:01:23Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCR [3.0561992956541606]
  We first present a comprehensive literature review of previous work, covering key facets of AI software testing processes. We then introduce a 3D classification model to systematically evaluate the image-based text extraction AI function. To evaluate the performance of our proposed AI software quality test, we propose four evaluation metrics to cover different aspects.
  arXiv  Detail & Related papers  (2024-09-14T23:33:28Z)
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants [176.39275404745098]
  We evaluate whether two AI assistants, GPT-3.5 and GPT-4, can adequately answer assessment questions.<n>GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions.<n>Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
  arXiv  Detail & Related papers  (2024-08-07T12:11:49Z)
• A Survey of Artificial Intelligence in Gait-Based Neurodegenerative Disease Diagnosis [51.07114445705692]
  neurodegenerative diseases (NDs) traditionally require extensive healthcare resources and human effort for medical diagnosis and monitoring.<n>As a crucial disease-related motor symptom, human gait can be exploited to characterize different NDs.<n>The current advances in artificial intelligence (AI) models enable automatic gait analysis for NDs identification and classification.
  arXiv  Detail & Related papers  (2024-05-21T06:44:40Z)
• AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment [54.93996119324928]
  We create the largest AIGI subjective quality database to date with 20,000 AIGIs and 420,000 subjective ratings, known as AIGIQA-20K. We conduct benchmark experiments on this database to assess the correspondence between 16 mainstream AIGI quality models and human perception.
  arXiv  Detail & Related papers  (2024-04-04T12:12:24Z)
• Assessing test artifact quality -- A tertiary study [1.7827643249624088]
  We have carried out a systematic literature review to identify and analyze existing secondary studies on quality aspects of software testing artifacts. We present an aggregation of the context dimensions and factors that can be used to characterize the environment in which the test case/suite quality is investigated.
  arXiv  Detail & Related papers  (2024-02-14T19:31:57Z)
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment [66.91538273487379]
  There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data.
  arXiv  Detail & Related papers  (2023-02-13T16:44:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.