Synthetic social data: trials and tribulations

• URL: http://arxiv.org/abs/2510.19952v1
• Date: Wed, 22 Oct 2025 18:25:42 GMT
• Title: Synthetic social data: trials and tribulations
• Authors: Guido Ivetta, Laura Moradbakhti, Rafael A. Calvo,
• Abstract summary: We explore the statistical representation of social values across four countries for six Large Language Models.<n>We compare machine-generated outputs with actual human survey data.<n>Our findings suggest that, despite the logistical and financial constraints of human surveys, even a small, skewed sample of real respondents may provide more reliable insights.
• Score: 3.713365412512855
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models are being used in conversational agents that simulate human conversations and generate social studies data. While concerns about the models' biases have been raised and discussed in the literature, much about the data generated is still unknown. In this study we explore the statistical representation of social values across four countries (UK, Argentina, USA and China) for six LLMs, with equal representation for open and closed weights. By comparing machine-generated outputs with actual human survey data, we assess whether algorithmic biases in LLMs outweigh the biases inherent in real- world sampling, including demographic and response biases. Our findings suggest that, despite the logistical and financial constraints of human surveys, even a small, skewed sample of real respondents may provide more reliable insights than synthetic data produced by LLMs. These results highlight the limitations of using AI-generated text for social research and emphasize the continued importance of empirical human data collection.

Related papers

• Who are you, ChatGPT? Personality and Demographic Style in LLM-Generated Content [5.515596385935823]
  Generative large language models (LLMs) have become central to everyday life, producing human-like text across diverse domains.<n>A growing body of research investigates whether these models also exhibit personality- and demographic-like characteristics in their language.<n>We introduce a novel, data-driven methodology for assessing LLM personality without relying on self-report questionnaires.<n>Applying instead automatic personality and gender classifiers to model replies on open-ended questions collected from Reddit.
  arXiv  Detail & Related papers  (2025-10-13T14:06:17Z)
• Population-Aligned Persona Generation for LLM-based Social Simulation [58.84363795421489]
  We propose a systematic framework for synthesizing high-quality, population-aligned persona sets for social simulation.<n>Our approach begins by leveraging large language models to generate narrative personas from long-term social media data.<n>To address the needs of specific simulation contexts, we introduce a task-specific module that adapts the globally aligned persona set to targeted subpopulations.
  arXiv  Detail & Related papers  (2025-09-12T10:43:47Z)
• Valid Inference with Imperfect Synthetic Data [39.10587411316875]
  We introduce a new estimator based on generalized method of moments.<n>We find that interactions between the moment residuals of synthetic data and those of real data can greatly improve estimates of the target parameter.
  arXiv  Detail & Related papers  (2025-08-08T18:32:52Z)
• Human Preferences in Large Language Model Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Prediction [5.774786149181393]
  We analyze how demographic attributes and prompt variations influence latent opinion mappings in large language models (LLMs)<n>We find that LLM-generated data fails to replicate the variance observed in real-world human responses.<n>In the political space, persona-to-party mappings exhibit limited differentiation, resulting in synthetic data that lacks the nuanced distribution of opinions found in survey data.
  arXiv  Detail & Related papers  (2025-02-22T16:25:33Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions.<n>As a testbed, we use country-level results from two global cultural surveys.<n>We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs? [1.7819574476785418]
  This study explores the potential of Large Language Models (LLMs) to generate artificial surveys.<n>By leveraging LLMs for synthetic data creation, we aim to address the limitations of traditional survey methods.<n>A novel approach incorporating "Personas" is introduced and compared to five other synthetic survey methods.
  arXiv  Detail & Related papers  (2025-01-20T15:11:03Z)
• Large Language Models for Market Research: A Data-augmentation Approach [3.3199591445531453]
  Large Language Models (LLMs) have transformed artificial intelligence by excelling in complex natural language processing tasks.<n>Recent studies highlight a significant gap between LLM-generated and human data, with biases introduced when substituting between the two.<n>We propose a novel statistical data augmentation approach that efficiently integrates LLM-generated data with real data in conjoint analysis.
  arXiv  Detail & Related papers  (2024-12-26T22:06:29Z)
• Agentic Society: Merging skeleton from real world and texture from Large Language Model [4.740886789811429]
  This paper explores a novel framework that leverages census data and large language models to generate virtual populations. We show that our method produces personas with variability essential for simulating diverse human behaviors in social science experiments. But the evaluation result shows that only weak sign of statistical truthfulness can be produced due to limited capability of current LLMs.
  arXiv  Detail & Related papers  (2024-09-02T08:28:19Z)
• Social Debiasing for Fair Multi-modal LLMs [59.61512883471714]
  Multi-modal Large Language Models (MLLMs) have dramatically advanced the research field and delivered powerful vision-language understanding capabilities.<n>These models often inherit deep-rooted social biases from their training data, leading to uncomfortable responses with respect to attributes such as race and gender.<n>This paper addresses the issue of social biases in MLLMs by introducing a comprehensive counterfactual dataset with multiple social concepts.
  arXiv  Detail & Related papers  (2024-08-13T02:08:32Z)
• Data-Centric AI in the Age of Large Language Models [51.20451986068925]
  This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs) We make the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs. We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
  arXiv  Detail & Related papers  (2024-06-20T16:34:07Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.