Generative AI as a Safety Net for Survey Question Refinement

• URL: http://arxiv.org/abs/2509.08702v1
• Date: Wed, 10 Sep 2025 15:52:10 GMT
• Title: Generative AI as a Safety Net for Survey Question Refinement
• Abstract summary: ChatGPT and other generative AI model platforms have been integrated into everyday life processes.<n>We have implemented a prompt experiment to systematically test what kind of feedback on survey questions an average ChatGPT user can expect.<n>Results from our zero-shot prompt experiment, which randomized the version of ChatGPT and the persona given to the model, shows that generative AI is a valuable tool today, even for an average AI user.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Writing survey questions that easily and accurately convey their intent to a variety of respondents is a demanding and high-stakes task. Despite the extensive literature on best practices, the number of considerations to keep in mind is vast and even small errors can render collected data unusable for its intended purpose. The process of drafting initial questions, checking for known sources of error, and developing solutions to those problems requires considerable time, expertise, and financial resources. Given the rising costs of survey implementation and the critical role that polls play in media, policymaking, and research, it is vital that we utilize all available tools to protect the integrity of survey data and the financial investments made to obtain it. Since its launch in 2022, ChatGPT and other generative AI model platforms have been integrated into everyday life processes and workflows, particularly pertaining to text revision. While many researchers have begun exploring how generative AI may assist with questionnaire design, we have implemented a prompt experiment to systematically test what kind of feedback on survey questions an average ChatGPT user can expect. Results from our zero--shot prompt experiment, which randomized the version of ChatGPT and the persona given to the model, shows that generative AI is a valuable tool today, even for an average AI user, and suggests that AI will play an increasingly prominent role in the evolution of survey development best practices as precise tools are developed.

Related papers

• Hybrid Panels: Toward Human-AI Collaboration in Survey Research [9.837591528122095]
  Large-scale population surveys are essential for generating robust social and scientific insights.<n>Advances in artificial intelligence have opened up new opportunities for AI-supported survey infrastructures.<n>A promising AI-enabled survey infrastructure for which we build a first pilot is a hybrid panel.
  arXiv  Detail & Related papers  (2026-08-23T20:27:48Z)
• Useful for Exploration, Risky for Precision: Evaluating AI Tools in Academic Research [0.0]
  The present work proposes and applies a benchmarking framework combining human-centered and computer-centered metrics to evaluate AI-based Q&A and literature review tools for research use.<n>The findings suggest that Q&A tools can offer valuable overviews and generally accurate summaries; however, they are not always reliable for precise information extraction.<n>The findings underscore the importance of explainability features to enhance transparency, verification efficiency and careful integration of AI tools into researchers.
  arXiv  Detail & Related papers  (2026-05-11T07:39:41Z)
• AIDABench: AI Data Analytics Benchmark [62.45908988324612]
  AIDABench is a benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner.<n>AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation.<n>We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families.
  arXiv  Detail & Related papers  (2026-02-27T08:58:05Z)
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• Stop DDoS Attacking the Research Community with AI-Generated Survey Papers [27.03595915182805]
  We argue that we must stop uploading massive amounts of AI-generated survey papers to the research community.<n>We call for restoring expert oversight and transparency in AI usage.<n>We show that safeguarding the integrity of surveys is no longer optional but imperative to the research community.
  arXiv  Detail & Related papers  (2025-10-09T05:55:13Z)
• Barbarians at the Gate: How AI is Upending Systems Research [58.95406995634148]
  We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery.<n>We term this approach as AI-Driven Research for Systems ( ADRS), which iteratively generates, evaluates, and refines solutions.<n>Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI.
  arXiv  Detail & Related papers  (2025-10-07T17:49:24Z)
• Information Retrieval in the Age of Generative AI: The RGB Model [77.96475639967431]
  This paper presents a novel quantitative approach to shed light on the complex information dynamics arising from the growing use of generative AI tools.<n>We propose a model to characterize the generation, indexing, and dissemination of information in response to new topics.<n>Our findings suggest that the rapid pace of generative AI adoption, combined with increasing user reliance, can outpace human verification, escalating the risk of inaccurate information proliferation.
  arXiv  Detail & Related papers  (2025-04-29T10:21:40Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions.<n>As a testbed, we use country-level results from two global cultural surveys.<n>We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• Exploring the Potential Role of Generative AI in the TRAPD Procedure for Survey Translation [0.0]
  This paper explores and assesses in what ways generative AI can assist in translating survey instruments.<n>We implement a zero-shot prompt experiment using ChatGPT to explore generative AI's ability to identify features of questions that might be difficult to translate to a linguistic audience.
  arXiv  Detail & Related papers  (2024-11-18T20:53:58Z)
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent [92.5712549836791]
  Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs)<n>We propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch.
  arXiv  Detail & Related papers  (2024-11-05T09:27:21Z)
• AutoSurvey: Large Language Models Can Automatically Write Surveys [77.0458309675818]
  This paper introduces AutoSurvey, a speedy and well-organized methodology for automating the creation of comprehensive literature surveys. Traditional survey paper creation faces challenges due to the vast volume and complexity of information. Our contributions include a comprehensive solution to the survey problem, a reliable evaluation method, and experimental validation demonstrating AutoSurvey's effectiveness.
  arXiv  Detail & Related papers  (2024-06-10T12:56:06Z)
• Detecting The Corruption Of Online Questionnaires By Artificial Intelligence [1.9458156037869137]
  This study tested if text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. Humans were able to correctly identify authorship of text above chance level. But their performance was still below what would be required to ensure satisfactory data quality.
  arXiv  Detail & Related papers  (2023-08-14T23:47:56Z)
• A Comprehensive Review of Trends, Applications and Challenges In Out-of-Distribution Detection [0.76146285961466]
  Field of study has emerged, focusing on detecting out-of-distribution data subsets and enabling a more comprehensive generalization. As many deep learning based models have achieved near-perfect results on benchmark datasets, the need to evaluate these models' reliability and trustworthiness is felt more strongly than ever. This paper presents a survey that, in addition to reviewing more than 70 papers in this field, presents challenges and directions for future works and offers a unifying look into various types of data shifts and solutions for better generalization.
  arXiv  Detail & Related papers  (2022-09-26T18:13:14Z)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers [66.11048565324468]
  We present a dataset of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers.
  arXiv  Detail & Related papers  (2021-05-07T00:12:34Z)
• Predicting respondent difficulty in web surveys: A machine-learning approach based on mouse movement features [3.6944296923226316]
  This paper explores the predictive value of mouse-tracking data with regard to respondents' difficulty. We use data from a survey on respondents' employment history and demographic information. We develop a personalization method that adjusts for respondents' baseline mouse behavior and evaluate its performance.
  arXiv  Detail & Related papers  (2020-11-05T10:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.