How Many Interviews Are Enough in a Software Engineering Study? Preliminary Findings on Sample Size and Saturation

• URL: http://arxiv.org/abs/2607.12326v1
• Date: Tue, 14 Jul 2026 04:15:51 GMT
• Title: How Many Interviews Are Enough in a Software Engineering Study? Preliminary Findings on Sample Size and Saturation
• Abstract summary: This paper investigates how interview sample adequacy and saturation are operationalized in empirical software engineering research.<n>We analyzed papers published between 2016 and 2025 across major software engineering venues.
• Score: 2.012425476229879
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background. Interview based studies are widely used in empirical software engineering to investigate human, organizational, and socio technical phenomena, yet interview sample adequacy and saturation are reported inconsistently across the literature. Aims. This paper investigates how interview sample adequacy and saturation are operationalized in empirical software engineering research. Method. We analyzed papers published between 2016 and 2025 across major software engineering venues, focusing on interview sample sizes, saturation discussions, and sample adequacy justifications. Results. Preliminary findings indicate substantial variation in sample sizes, from highly specialized small sample studies to broader investigations involving large interview datasets. Studies involving fewer than 12 interviewees were common and frequently associated with specialized industrial contexts or constrained organizational access. However, the most recurrent range was 13 to 24 interviewees, suggesting that moderate sized samples represent the most common configuration in empirical software engineering research. Saturation and sample adequacy justification were heterogeneous, with many studies relying on implicit or contextual reasoning rather than explicit methodological discussion. Conclusions. Our findings provide initial empirical insights into methodological reporting practices in interview based software engineering research and contribute to ongoing discussions regarding qualitative rigor and transparency.

Related papers

• The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys [3.183470571353323]
  Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys.<n>This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings.
  arXiv  Detail & Related papers  (2026-07-26T19:08:31Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers [52.97586643334384]
  We introduce PaperScope, a benchmark for agentic deep research.<n>It is built on a knowledge graph of over 2,000 AI papers spanning three years.<n>It integrates semantically related key information nodes and employs optimized random-walk article selector.
  arXiv  Detail & Related papers  (2026-04-13T11:07:08Z)
• Measuring the State of Open Science in Transportation Using Large Language Models [8.915048816245394]
  Open science initiatives have strengthened scientific integrity and accelerated research progress across many fields.<n>Key features of open science, defined here as data and code availability, are difficult to extract due to the inherent complexity of the field.<n>This paper introduces an automatic and scalable feature-extraction pipeline to measure data and code availability in transportation research.
  arXiv  Detail & Related papers  (2026-01-20T19:39:52Z)
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis [3.6324565773746147]
  We conduct a so-called multiverse analysis on a published empirical SE paper.<n>We identify nine pivotal analytical decisions with at least one equally defensible alternative.<n>The overwhelming majority produced qualitatively different, and sometimes even opposite, findings.
  arXiv  Detail & Related papers  (2025-12-09T18:47:00Z)
• Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination [77.69093448529455]
  We present an empirical study using an infinitely scalable framework to synthesize research-level QA directly from arXiv papers.<n>We evaluate a lack of significant performance decay near knowledge cutoff dates for models of various sizes, developers, and release dates.<n>We hypothesize that the multi-step reasoning required by our synthesis pipeline offered additional complexity that goes deeper than shallow memorization.
  arXiv  Detail & Related papers  (2025-08-26T16:41:37Z)
• The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis [51.046457649151336]
  We propose a perspective on interpretability research grounded in causal mediation analysis.<n>We describe the history and current state of interpretability taxonomized according to the types of causal units (mediators) employed.<n>We discuss the pros and cons of each mediator, providing insights as to when particular kinds of mediators and search methods are most appropriate.
  arXiv  Detail & Related papers  (2024-08-02T17:51:42Z)
• Meta-survey on outlier and anomaly detection [0.0]
  This paper implements the first systematic meta-survey of general surveys and reviews on outlier and anomaly detection. It collects nearly 500 papers using two specialized scientific search engines. The paper investigates the evolution of the outlier detection field over a 20-year period, revealing emerging themes and methods.
  arXiv  Detail & Related papers  (2023-12-12T09:29:22Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold [88.83876819883653]
  We show through a manual classification of recent NLP research papers that this is indeed the case. We observe that NLP research often goes beyond the square one setup, focusing not only on accuracy, but also on fairness or interpretability, but typically only along a single dimension.
  arXiv  Detail & Related papers  (2022-06-20T13:04:23Z)
• Exploring Context Generalizability in Citywide Crowd Mobility Prediction: An Analytic Framework and Benchmark [8.32409186492201]
  We present a unified analytic framework and a large-scale benchmark for evaluating context generalizability.<n>We conduct experiments in several crowd mobility prediction tasks such as bike flow, metro passenger flow, and electric vehicle charging demand.<n>Using more contextual features may not always result in better prediction with existing context modeling techniques.<n>In context modeling techniques, using a gated unit to incorporate raw contextual features into the deep prediction model has good generalizability.
  arXiv  Detail & Related papers  (2021-06-30T13:19:41Z)
• Topic Space Trajectories: A case study on machine learning literature [0.0]
  We present topic space trajectories, a structure that allows for the comprehensible tracking of research topics. We show the applicability of our approach on a publication corpus spanning 50 years of machine learning research from 32 publication venues. Our novel analysis method may be employed for paper classification, for the prediction of future research topics, and for the recommendation of fitting conferences and journals for submitting unpublished work.
  arXiv  Detail & Related papers  (2020-10-23T10:53:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.