Do LLMs Ask the Right Questions? Evaluating GPT-Generated Surveys as Instruments for Measuring Social Attitudes

• URL: http://arxiv.org/abs/2607.19211v1
• Date: Tue, 21 Jul 2026 15:41:07 GMT
• Title: Do LLMs Ask the Right Questions? Evaluating GPT-Generated Surveys as Instruments for Measuring Social Attitudes
• Abstract summary: We present a comparison between GPT-generated surveys and established survey baselines across three social domains.<n>Our results show that GPT-generated surveys capture the same dominant attitudinal divisions as human-designed instruments.
• Score: 14.002581483623219
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Understanding human beliefs and social attitudes often relies on carefully designed survey instruments. Recent work has suggested that large language models (LLMs) could automate parts of this process by generating surveys at scale, raising questions about the comparability of such instruments to literature-grounded, human-designed surveys. We present a controlled empirical comparison between GPT-generated surveys and established survey baselines across three social domains: climate change, immigration, and diversity, equity, and inclusion (DEI). GPT-generated surveys were produced using a fixed prompting framework enforcing a 3x3 structure over beliefs, perceptions, and behaviors, while human baselines were assembled from validated instruments to match survey length and construct coverage. We collected responses from U.S.-based participants, who completed both survey types, allowing direct within-subject comparison. We analyze differences in response distributions, clustering behavior, and alignment with self-identified stances. Our results show that GPT-generated surveys capture the same dominant attitudinal divisions as human-designed instruments, while exhibiting differences in the resolution of belief structure and group separation. These findings suggest that LLM-generated surveys are suited for exploratory and large-scale analyses, and can be used to complement expert-designed instruments.

Related papers

• Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models [6.18279552770494]
  Large Language models (LLMs) have been trained on vast amounts of human-generated data.<n>LLMs show promise in mimicking human-like patterns that facilitate their use in simulating people in a variety of contexts.<n>One such context is using LLMs as'silicon samples', i.e., proxies of people in answering survey questions to establish public opinion, design policies, or use as (social) scientific data.
  arXiv  Detail & Related papers  (2026-09-09T14:59:07Z)
• SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators [50.129606229339146]
  SurveyReview is a reviewer-aligned, multi-dimensional benchmark and dataset for survey evaluation.<n>We structure authentic peer-review reports by converting free-form comments into four-dimensional scores.<n>We release standardized train/test splits and an evaluation protocol to measure alignment between automatic evaluators and human reviewers.
  arXiv  Detail & Related papers  (2026-08-07T16:11:46Z)
• On the Credibility of Evaluating LLMs using Survey Questions [0.42061757959666934]
  Recent studies evaluate the value orientation of large language models (LLMs) using adapted social surveys.<n>This paper identifies limitations in this methodology that, depending on the exact setup, can lead to both underestimating and overestimating the similarity of value orientation.<n>Using the World Value Survey in three languages across five countries, we demonstrate that prompting methods (direct vs. chain-of-thought) and decoding strategies (greedy vs. sampling) significantly affect results.
  arXiv  Detail & Related papers  (2026-02-03T21:45:43Z)
• AI Assisted Economics Measurement From Survey: Evidence from Public Employee Pension Choice [2.2324302196460706]
  We develop an iterative framework for economic measurement that leverages large language models to extract measurement structure directly from survey instruments.<n>The framework is applied to a large scale public employee retirement plan survey.
  arXiv  Detail & Related papers  (2026-02-02T02:22:36Z)
• AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys [14.699937408707356]
  We introduce AlignSurvey, the first benchmark that systematically replicates and evaluates the full social survey pipeline.<n>It defines four tasks aligned with key survey stages: social role modeling, semi-structured interview modeling, attitude stance modeling and survey response modeling.<n>It also provides task-specific evaluation metrics to assess alignment fidelity, consistency, and fairness at both individual and group levels.
  arXiv  Detail & Related papers  (2025-11-11T06:14:21Z)
• Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble [46.82793004650415]
  Large language models (LLMs) have demonstrated promise in emulating human-like responses across a range of tasks.<n>We propose a novel alignment framework that treats LLMs as agent proxies for human survey respondents.<n>We introduce P2P, a system that steers LLM agents toward representative behavioral patterns using structured prompt engineering, entropy-based sampling, and regression-based selection.
  arXiv  Detail & Related papers  (2025-09-14T15:08:45Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions.<n>As a testbed, we use country-level results from two global cultural surveys.<n>We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• ChatGPT vs Social Surveys: Probing Objective and Subjective Silicon Population [7.281887764378982]
  Large Language Models (LLMs) have the potential to simulate human responses in social surveys and generate reliable predictions.<n>We employ repeated random sampling to create sampling distributions that identify the population parameters of silicon samples generated by GPT.<n>Our findings show that GPT's demographic distribution aligns with the 2020 U.S. population in terms of gender and average age.<n> GPT's point estimates for attitudinal scores are highly inconsistent and show no clear inclination toward any particular ideology.
  arXiv  Detail & Related papers  (2024-09-04T10:33:37Z)
• Evaluating Large Language Models with Psychometrics [59.821829073478376]
  This paper offers a comprehensive benchmark for quantifying psychological constructs of Large Language Models (LLMs)<n>Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets.<n>We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy [66.95501113584541]
  We propose an Iterative utiliTy judgm fraEntMework (ITEM) to promote each step in Retrieval-Augmented Generation (RAG)<n>RAG's three core components -- relevance ranking derived from retrieval models, utility judgments, and answer generation -- align with Schutz's philosophical system of relevances.<n> Experimental results demonstrate significant improvements of ITEM in utility judgments, ranking, and answer generation upon representative baselines.
  arXiv  Detail & Related papers  (2024-06-17T07:52:42Z)
• An unsupervised learning approach to evaluate questionnaire data -- what one can learn from violations of measurement invariance [2.4762962548352467]
  This paper promotes an unsupervised learning-based approach to such research data. It works in three phases: data preparation, clustering of questionnaires, and measuring similarity based on the obtained clustering and the properties of each group. It provides a natural comparison between groups and a natural description of the response patterns of the groups.
  arXiv  Detail & Related papers  (2023-12-11T11:31:41Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction [0.0]
  Large language models (LLMs) that produce human-like responses have begun to revolutionize research practices in the social sciences. We develop a novel methodological framework that fine-tunes LLMs with repeated cross-sectional surveys. We introduce two new applications of the AI-augmented survey: retrodiction (i.e., predict year-level missing responses) and unasked opinion prediction.
  arXiv  Detail & Related papers  (2023-05-16T17:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.