AI-Conducted Interviews in Empirical Software Engineering: An Experience Report

• URL: http://arxiv.org/abs/2607.14452v1
• Date: Thu, 16 Jul 2026 00:59:44 GMT
• Title: AI-Conducted Interviews in Empirical Software Engineering: An Experience Report
• Abstract summary: Semi-structured interviews are resource-intensive and difficult to coordinate across schedules, locations, and natural languages.<n>This report examines a customized MyGPT used to conduct short, self-administered interviews in two ESE studies.
• Score: 1.4278764160724604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semi-structured interviews are widely used in empirical software engineering (ESE), but they are resource-intensive and difficult to coordinate across schedules, locations, and natural languages. This experience report examines a customized MyGPT used to conduct short, self-administered interviews in two ESE studies: one on refactoring practices and another on generative AI in Scrum-related activities. Participants accessed the interviewer through shared links, used voice interaction, selected a preferred natural language, and completed the interview without a researcher present. The AI followed a predefined protocol and generated a structured synthesis that participants voluntarily submitted; these artifacts were not treated as verbatim transcripts. We analyzed 66 submissions and questionnaire responses, and audited artifact format, language, length, and protocol consistency. Of the submitted artifacts, 92.4% followed the expected synthesis format, 65 were predominantly in Portuguese and one in English, and two conflicted with the reported protocol. Participants generally rated the experience positively: 90.9% reported a positive overall experience and comfort, 95.5% considered the questions clear, 97.0% rated the pace positively, and 89.4% would participate again. Reported limitations included generic questions, limited sensitivity to answers, insufficient depth, privacy concerns, and missed human interaction. The findings support the operational viability and acceptability of this workflow among analyzed respondents, but do not establish completion rates, time savings, summary fidelity, or equivalence to human-conducted interviews. AI interviewers should therefore be treated as a complementary option for short, focused, low-risk studies, with protocol design, privacy guidance, artifact validation, and human oversight.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• AI-assisted Script Management for Requirements Elicitation Interviews [0.4376619220604389]
  This paper presents an AI-assisted elicitation workflow that combines theory-guided script generation grounded in business goals with live support for topic coverage tracking and on-demand follow-up question generation.<n>Based on a rubric derived from elicitation best practices, the AI-generated scripts score higher than training-only scripts.
  arXiv  Detail & Related papers  (2026-08-03T03:22:37Z)
• SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery [55.50580661343875]
  We introduce SparkMe, a multi-agent interviewer that performs deliberative planning via simulated conversation rollouts to select questions with high expected utility.<n>We evaluate SparkMe through controlled experiments with LLM-based interviewees, showing that it achieves higher interview utility.<n>We further validate SparkMe in a user study with 70 participants across 7 professions on the impact of AI on their professions.
  arXiv  Detail & Related papers  (2026-02-24T17:33:02Z)
• Modular AI-Powered Interviewer with Dynamic Question Generation and Expertise Profiling [0.7349727826230863]
  This study presents an AI-powered interviewer that dynamically generates questions that are contextually appropriate and expertise aligned.<n>The interviewer is built on a locally hosted large language model (LLM) that generates coherent dialogue while preserving data privacy.<n>The proposed interviewer is a scalable, privacy-conscious solution that advances AI-assisted qualitative data collection.
  arXiv  Detail & Related papers  (2025-11-21T18:25:26Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• MimiTalk: Revolutionizing Qualitative Research with Dual-Agent AI [1.8219577154655007]
  We present MimiTalk, a dual-agent constitutional AI framework designed for scalable and ethical conversational data collection in social science research.<n>We conducted three studies: Study 1 evaluated usability with 20 participants; Study 2 compared 121 AI interviews to 1,271 human interviews from the MediaSum dataset using NLP metrics and propensity score matching; Study 3 involved 10 interdisciplinary researchers conducting both human and AI interviews, followed by blind thematic analysis.<n>Results indicate that MimiTalk reduces interview anxiety, maintains conversational coherence, and outperforms human interviews in information richness, coherence, and stability.
  arXiv  Detail & Related papers  (2025-09-27T16:02:50Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• LLMREI: Automating Requirements Elicitation Interviews with LLMs [47.032121951473435]
  This study introduces LLMREI, a chat bot designed to conduct requirements elicitation interviews with minimal human intervention.<n>We evaluated its performance in 33 simulated stakeholder interviews.<n>Our findings indicate that LLMREI makes a similar number of errors compared to human interviewers, is capable of extracting a large portion of requirements, and demonstrates a notable ability to generate highly context-dependent questions.
  arXiv  Detail & Related papers  (2025-07-03T12:18:05Z)
• AI-Assisted Conversational Interviewing: Effects on Data Quality and User Experience [0.0]
  This study introduces a framework for AI-assisted conversational interviewing.<n>We conducted a web survey experiment where 1,800 participants were randomly assigned to AI 'chatbots'<n>Our findings reveal the feasibility of using AI methods such as chatbots to enhance open-ended data collection in web surveys.
  arXiv  Detail & Related papers  (2025-04-09T13:58:07Z)
• AI Conversational Interviewing: Transforming Surveys with LLMs as Adaptive Interviewers [40.80290002598963]
  This study explores the potential of replacing human interviewers with large language models (LLMs) to conduct scalable conversational interviews.<n>We conducted a small-scale, in-depth study with university students who were randomly assigned to a conversational interview by either AI or human interviewers.<n>Various quantitative and qualitative measures assessed interviewer adherence to guidelines, response quality, participant engagement, and overall interview efficacy.
  arXiv  Detail & Related papers  (2024-09-16T16:03:08Z)
• INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions [47.90088587508672]
  InSCIt is a dataset for Information-Seeking Conversations with mixed-initiative Interactions. It contains 4.7K user-agent turns from 805 human-human conversations. We report results of two systems based on state-of-the-art models of conversational knowledge identification and open-domain question answering.
  arXiv  Detail & Related papers  (2022-07-02T06:18:12Z)
• Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots [51.091235903442715]
  This paper makes an attempt to explore the impact of utilizing personas that describe either self or partner speakers on the task of response selection. Four persona fusion strategies are designed, which assume personas interact with contexts or responses in different ways. Empirical studies on the Persona-Chat dataset show that the partner personas can improve the accuracy of response selection.
  arXiv  Detail & Related papers  (2021-05-19T10:32:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.