Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation

• URL: http://arxiv.org/abs/2509.06337v1
• Date: Mon, 08 Sep 2025 04:59:00 GMT
• Title: Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation
• Authors: Jianpeng Zhao, Chenyu Yuan, Weiming Luo, Haoling Xie, Guangwei Zhang, Steven Jige Quan, Zixuan Yuan, Pengyang Wang, Denghui Zhang,
• Abstract summary: This paper explores a new paradigm: simulating virtual survey respondents using Large Language Models (LLMs)<n>We introduce two novel simulation settings, namely Partial Attribute Simulation (PAS) and Full Attribute Simulation (FAS)<n>We curate a comprehensive benchmark suite, LLM-S3 (Large Language Model-based Sociodemographic Simulation Survey), that spans 11 real-world public datasets across four sociological domains.
• Score: 18.225151370273093
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Questionnaire-based surveys are foundational to social science research and public policymaking, yet traditional survey methods remain costly, time-consuming, and often limited in scale. This paper explores a new paradigm: simulating virtual survey respondents using Large Language Models (LLMs). We introduce two novel simulation settings, namely Partial Attribute Simulation (PAS) and Full Attribute Simulation (FAS), to systematically evaluate the ability of LLMs to generate accurate and demographically coherent responses. In PAS, the model predicts missing attributes based on partial respondent profiles, whereas FAS involves generating complete synthetic datasets under both zero-context and context-enhanced conditions. We curate a comprehensive benchmark suite, LLM-S^3 (Large Language Model-based Sociodemographic Survey Simulation), that spans 11 real-world public datasets across four sociological domains. Our evaluation of multiple mainstream LLMs (GPT-3.5/4 Turbo, LLaMA 3.0/3.1-8B) reveals consistent trends in prediction performance, highlights failure modes, and demonstrates how context and prompt design impact simulation fidelity. This work establishes a rigorous foundation for LLM-driven survey simulations, offering scalable and cost-effective tools for sociological research and policy evaluation. Our code and dataset are available at: https://github.com/dart-lab-research/LLM-S-Cube-Benchmark

Related papers

• Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data [54.145424717168794]
  Large Language Models (LLMs) have demonstrated remarkable human-like capabilities, yet their ability to replicate a specific individual remains under-explored.<n>This paper presents a case study to investigate LLM-based individual simulation with a volunteer-contributed archive of private messaging history spanning over ten years.<n>We propose the "Individual Turing Test" to evaluate whether acquaintances of the volunteer can correctly identify which response in a multi-candidate pool most plausibly comes from the volunteer.
  arXiv  Detail & Related papers  (2026-03-01T21:46:27Z)
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
  We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
  arXiv  Detail & Related papers  (2025-10-20T13:14:38Z)
• Population-Aligned Persona Generation for LLM-based Social Simulation [58.84363795421489]
  We propose a systematic framework for synthesizing high-quality, population-aligned persona sets for social simulation.<n>Our approach begins by leveraging large language models to generate narrative personas from long-term social media data.<n>To address the needs of specific simulation contexts, we introduce a task-specific module that adapts the globally aligned persona set to targeted subpopulations.
  arXiv  Detail & Related papers  (2025-09-12T10:43:47Z)
• Can Large Language Models Simulate Human Responses? A Case Study of Stated Preference Experiments in the Context of Heating-related Choices [2.2582258282563763]
  Stated preference (SP) surveys are a key method to research how individuals make trade-offs in hypothetical, also futuristic, scenarios.<n>They tend to be costly, time-consuming, and can be affected by respondent fatigue and ethical constraints.<n>This study investigates the use of large language models (LLMs) to simulate consumer choices in energy-related surveys.
  arXiv  Detail & Related papers  (2025-03-07T10:37:31Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions.<n>As a testbed, we use country-level results from two global cultural surveys.<n>We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents [47.935533238820334]
  Traditional sociological research often relies on human participation, which, though effective, is expensive, challenging to scale, and with ethical concerns.<n>Recent advancements in large language models (LLMs) highlight their potential to simulate human behavior, enabling the replication of individual responses and facilitating studies on many interdisciplinary studies.<n>We categorize the simulations into three types: (1) Individual Simulation, which mimics specific individuals or demographic groups; (2) Scenario Simulation, where multiple agents collaborate to achieve goals within specific contexts; and (3) Simulation Society, which models interactions within agent societies to reflect the complexity and variety of real-world dynamics.
  arXiv  Detail & Related papers  (2024-12-04T18:56:37Z)
• GenSim: A General Social Simulation Platform with Large Language Model based Agents [111.00666003559324]
  We propose a novel large language model (LLMs)-based simulation platform called textitGenSim.<n>Our platform supports one hundred thousand agents to better simulate large-scale populations in real-world contexts.<n>To our knowledge, GenSim represents an initial step toward a general, large-scale, and correctable social simulation platform.
  arXiv  Detail & Related papers  (2024-10-06T05:02:23Z)
• Agentic Society: Merging skeleton from real world and texture from Large Language Model [4.740886789811429]
  This paper explores a novel framework that leverages census data and large language models to generate virtual populations. We show that our method produces personas with variability essential for simulating diverse human behaviors in social science experiments. But the evaluation result shows that only weak sign of statistical truthfulness can be produced due to limited capability of current LLMs.
  arXiv  Detail & Related papers  (2024-09-02T08:28:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.