A Methodological Framework for LLM-Based Mining of Software Repositories

• URL: http://arxiv.org/abs/2508.02233v1
• Date: Mon, 04 Aug 2025 09:33:47 GMT
• Title: A Methodological Framework for LLM-Based Mining of Software Repositories
• Authors: Vincenzo De Martino, Joel Castaño, Fabio Palomba, Xavier Franch, Silverio Martínez-Fernández,
• Abstract summary: Large Language Models (LLMs) are increasingly used in software engineering research.<n>Despite their growing popularity, the methodological integration of LLMs into Mining Software Repositories (MSR) remains poorly understood.
• Score: 12.504438766461027
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly used in software engineering research, offering new opportunities for automating repository mining tasks. However, despite their growing popularity, the methodological integration of LLMs into Mining Software Repositories (MSR) remains poorly understood. Existing studies tend to focus on specific capabilities or performance benchmarks, providing limited insight into how researchers utilize LLMs across the full research pipeline. To address this gap, we conduct a mixed-method study that combines a rapid review and questionnaire survey in the field of LLM4MSR. We investigate (1) the approaches and (2) the threats that affect the empirical rigor of researchers involved in this field. Our findings reveal 15 methodological approaches, nine main threats, and 25 mitigation strategies. Building on these findings, we present PRIMES 2.0, a refined empirical framework organized into six stages, comprising 23 methodological substeps, each mapped to specific threats and corresponding mitigation strategies, providing prescriptive and adaptive support throughout the lifecycle of LLM-based MSR studies. Our work contributes to establishing a more transparent and reproducible foundation for LLM-based MSR research.

Related papers

• A Survey of AIOps in the Era of Large Language Models [60.59720351854515]
  We analyzed 183 research papers published between January 2020 and December 2024 to answer four key research questions (RQs)<n>We discuss the state-of-the-art advancements and trends, identify gaps in existing research, and propose promising directions for future exploration.
  arXiv  Detail & Related papers  (2025-06-23T02:40:16Z)
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning [87.30285670315334]
  textbfR1-Searcher is a novel two-stage outcome-based RL approach designed to enhance the search capabilities of Large Language Models.<n>Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start.<n>Our experiments demonstrate that our method significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini.
  arXiv  Detail & Related papers  (2025-03-07T17:14:44Z)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search [57.28671084993782]
  Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains.<n>Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities.<n>We propose a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-04T17:26:58Z)
• A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering [12.504438766461027]
  Large Language Models (LLMs) have transformed Software Engineering (SE) by providing innovative methods for analyzing software repositories.<n>Our research packages a framework, coined Prompt Refinement and Insights for Mining Empirical Software repositories (PRIMES)<n>Our findings indicate that standardizing prompt engineering and using PRIMES can enhance the reliability and accuracy of studies utilizing LLMs.
  arXiv  Detail & Related papers  (2024-11-15T06:08:57Z)
• EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration [76.66831821738927]
  Large language models (LLMs) remain under-studied in scenarios requiring optimal decision-making under uncertainty.<n>We measure LLMs' (in)ability to make optimal decisions in bandits, a state-less reinforcement learning setting relevant to many applications.<n>Motivated by the existence of optimal exploration algorithms, we propose efficient ways to integrate this algorithmic knowledge into LLMs.
  arXiv  Detail & Related papers  (2024-10-08T17:54:03Z)
• Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning [0.9110413356918055]
  This research pioneers the use of fine-tuned Large Language Models (LLMs) to automate Systematic Literature Reviews ( SLRs) Our study employed the latest fine-tuning methodologies together with open-sourced LLMs, and demonstrated a practical and efficient approach to automating the final execution stages of an SLR process. The results maintained high fidelity in factual accuracy in LLM responses, and were validated through the replication of an existing PRISMA-conforming SLR.
  arXiv  Detail & Related papers  (2024-04-08T00:08:29Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Large Language Models for Software Engineering: A Systematic Literature Review [34.12458948051519]
  Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE) We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs) From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study.
  arXiv  Detail & Related papers  (2023-08-21T10:37:49Z)
• Information Extraction in Low-Resource Scenarios: Survey and Perspective [56.5556523013924]
  Information Extraction seeks to derive structured information from unstructured texts. This paper presents a review of neural approaches to low-resource IE from emphtraditional and emphLLM-based perspectives.
  arXiv  Detail & Related papers  (2022-02-16T13:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.