The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys

• URL: http://arxiv.org/abs/2607.23805v1
• Date: Sun, 26 Jul 2026 19:08:31 GMT
• Title: The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys
• Abstract summary: Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys.<n>This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings.
• Score: 3.183470571353323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys. Aims: This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings. Method: We conducted a secondary analysis of four software engineering survey datasets using manual identification of suspicious responses, automated AI generated text detection, descriptive statistical analysis, and thematic analysis. We compared findings obtained from the original and manually cleaned datasets. Results: Quantitative findings generally remained stable after filtering suspicious responses, although some demographic and analytical variables showed moderate variation, affecting the interpretation of specific participant groups and contextual characteristics. In contrast, qualitative findings were more strongly influenced by changes in contextual framing, code prominence, and the nature of the evidence supporting interpretation, shaping how participants' experiences and study contexts were interpreted and characterized. Conclusions: AI assisted participation may influence software engineering survey findings differently depending on the type of analysis being conducted. The findings reinforce the importance of combining multiple validation procedures, particularly in studies relying on open ended responses.

Related papers

• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse [22.927943525772857]
  We show that fully autonomous AI analysts built on large language models (LLMs) can reproduce a similar structured analytic diversity cheaply and at scale.<n>We show that the effects are emphsteerable: reassigning the analyst persona or LLM shifts the distribution of outcomes even after excluding methodologically deficient runs.
  arXiv  Detail & Related papers  (2026-02-21T04:10:21Z)
• Human-AI Collaborative Inductive Thematic Analysis: AI Guided Analysis and Human Interpretive Authority [0.750605871308538]
  This study examines how researchers interact with an Inductive Thematic Analysis GPT (ITA-GPT)<n>ITA-GPT is a purpose-built AI tool designed to support inductive thematic analysis.<n>The study focuses on analytic process rather than substantive findings.
  arXiv  Detail & Related papers  (2026-01-17T00:38:36Z)
• An Investigation on How AI-Generated Responses Affect SoftwareEngineering Surveys [3.183470571353323]
  This study explores how large language models (LLMs) are being misused in software engineering surveys.<n>We analyzed data from two survey deployments conducted in 2025 through the Prolific platform.<n>We identify data authenticity as an emerging dimension of validity in software engineering surveys.
  arXiv  Detail & Related papers  (2025-12-19T11:17:05Z)
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research [81.04845910798387]
  Generating natural language explanations for threat detections remains an open problem in cybersecurity research.<n>We present AutoMalDesc, an automated static analysis summarization framework that operates independently at scale.<n>We publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) datasets, along with our methodology and evaluation framework.
  arXiv  Detail & Related papers  (2025-11-17T13:05:25Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• Methodological Foundations for AI-Driven Survey Question Generation [41.94295877935867]
  This paper presents a methodological framework for using generative AI in educational survey research.<n>We explore how Large Language Models can generate adaptive, context-aware survey questions.<n>We examine ethical issues such as bias, privacy, and transparency.
  arXiv  Detail & Related papers  (2025-05-02T09:50:34Z)
• Dataset resulting from the user study on comprehensibility of explainable AI algorithms [5.775094401949666]
  This paper introduces a dataset that is the result of a user study on the comprehensibility of explainable artificial intelligence (XAI) algorithms.<n>The study participants were recruited from 149 candidates to form three groups representing experts in the domain of mycology (DE)<n>The main part of the dataset contains 39 transcripts of interviews during which participants were asked to complete a series of tasks and questions related to the interpretation of decisions of a machine learning model trained to distinguish between edible and inedible mushrooms.
  arXiv  Detail & Related papers  (2024-10-21T11:37:58Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• Sentiment Analysis Based on Deep Learning: A Comparative Study [69.09570726777817]
  The study of public opinion can provide us with valuable information. The efficiency and accuracy of sentiment analysis is being hindered by the challenges encountered in natural language processing. This paper reviews the latest studies that have employed deep learning to solve sentiment analysis problems.
  arXiv  Detail & Related papers  (2020-06-05T16:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.