Another Systematic Review? A Critical Analysis of Systematic Literature Reviews on Agile Effort and Cost Estimation

• URL: http://arxiv.org/abs/2601.20893v1
• Date: Wed, 28 Jan 2026 08:07:28 GMT
• Title: Another Systematic Review? A Critical Analysis of Systematic Literature Reviews on Agile Effort and Cost Estimation
• Authors: Henry Edison, Nauman Ali,
• Abstract summary: Systematic literature reviews ( SLRs) have become prevalent in software engineering research.<n>The proliferation of overlapping and often repetitive SLRs indicates that researchers are not extensively checking for existing SLRs on a topic.<n>We identify common justification patterns through a qualitative content analysis of 18 published SLRs.
• Score: 0.08594140167290097
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Background: Systematic literature reviews (SLRs) have become prevalent in software engineering research. Several researchers may conduct SLRs on similar topics without a prospective register for SLR protocols. However, even ignoring these unavoidable duplications of effort in the simultaneous conduct of SLRs, the proliferation of overlapping and often repetitive SLRs indicates that researchers are not extensively checking for existing SLRs on a topic. Given how effort-intensive it is to design, conduct, and report an SLR, the situation is less than ideal for software engineering research. Aim: To understand how authors justify additional SLRs on a topic. Method: To illustrate the issue and develop suggestions for improvement to address this issue, we have intentionally picked a sufficiently narrow but well-researched topic, i.e., effort estimation in Agile software development. We identify common justification patterns through a qualitative content analysis of 18 published SLRs. We further consider the citation data, publication years, publication venues, and the quality of the SLRs when interpreting the results. Results: The common justification patterns include authors claiming gaps in coverage, methodological limitations in prior studies, temporal obsolescence of previous SLRs, or rapid technological and methodological advancements necessitating updated syntheses. Conclusion: Our in-depth analysis of SLRs on a fairly narrow topic provides insights into SLRs in software engineering in general. By emphasizing the need for identifying existing SLRs and for justifying the undertaking of further SLRs, both in design and review guidelines and as a policy of conferences and journals, we can reduce the likelihood of duplication of effort and increase the rate of progress in the field.

Related papers

• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• SoK: Potentials and Challenges of Large Language Models for Reverse Engineering [5.603029122508333]
  Reverse Engineering (RE) is central to software security, enabling tasks such as vulnerability discovery and malware analysis.<n>Earlier advances in deep learning start to automate parts of RE, particularly for malware detection and vulnerability classification.<n>More recently, a rapidly growing body of work has applied Large Language Models (LLMs) to similar purposes.
  arXiv  Detail & Related papers  (2025-09-26T03:26:51Z)
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs [69.10441885629787]
  Retrieval-Augmented Generation (RAG) lifts the factuality of Large Language Models (LLMs) by injecting external knowledge.<n>It falls short on problems that demand multi-step inference; conversely, purely reasoning-oriented approaches often hallucinate or mis-ground facts.<n>This survey synthesizes both strands under a unified reasoning-retrieval perspective.
  arXiv  Detail & Related papers  (2025-07-13T03:29:41Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• Leveraging Sustainable Systematic Literature Reviews [8.18445480530188]
  This paper presents concrete directions towards sustainable SLRs.<n>We first identified 18 green drivers'' (GD) that could directly impact SLR sustainability.<n>We distilled 25 sustainability indicators (SI) associated with the GD to assess SLRs regarding their sustainability.
  arXiv  Detail & Related papers  (2025-01-03T14:03:15Z)
• RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement [85.08223786819532]
  Existing large language models (LLMs) show exceptional problem-solving capabilities but might struggle with complex reasoning tasks.<n>We propose textbfRAG-Star, a novel RAG approach that integrates retrieved information to guide the tree-based deliberative reasoning process.<n>Our experiments involving Llama-3.1-8B-Instruct and GPT-4o demonstrate that RAG-Star significantly outperforms previous RAG and reasoning methods.
  arXiv  Detail & Related papers  (2024-12-17T13:05:36Z)
• Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning [52.83539473110143]
  We introduce a novel structure-oriented analysis method to help Large Language Models (LLMs) better understand a question. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA) Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods.
  arXiv  Detail & Related papers  (2024-10-18T05:30:33Z)
• System for systematic literature review using multiple AI agents: Concept and an empirical evaluation [3.453564255183234]
  Systematic literature review ( SLR) is foundational to evidence-based research.<n>We introduce a novel multi-AI-agent system designed to fully automate SLRs.<n>We show how it substantially reduces the time and effort traditionally required for SLRs while maintaining comprehensiveness and precision.
  arXiv  Detail & Related papers  (2024-03-13T10:27:52Z)
• Emerging Results on Automated Support for Searching and Selecting Evidence for Systematic Literature Review Updates [1.1153433121962064]
  We present emerging results on an automated approach to support searching and selecting studies for SLR updates in Software Engineering. We developed an automated tool prototype to perform the snowballing search technique and support selecting relevant studies for SLR updates using Machine Learning (ML) algorithms.
  arXiv  Detail & Related papers  (2024-02-07T23:39:20Z)
• How Many Papers Should You Review? A Research Synthesis of Systematic Literature Reviews in Software Engineering [5.6292136785289175]
  We aim to provide more understanding of when an SLR in Software Engineering should be conducted. A research synthesis was conducted on a sample of 170 SLRs published in top-tier SE journals. The results of our study can be used by SE researchers as an indicator or benchmark to understand whether an SLR is conducted at a good time.
  arXiv  Detail & Related papers  (2023-07-12T10:18:58Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)
• A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets [0.0]
  SLR usage has increased in many applications, but the environment, background image resolution, modalities, and datasets affect the performance a lot. This review paper facilitates a comprehensive overview of SLR and discusses the needs, challenges, and problems associated with SLR. Research progress and existing state-of-the-art SLR models over the past decade have been reviewed.
  arXiv  Detail & Related papers  (2022-04-07T09:49:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.