When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

• URL: http://arxiv.org/abs/2607.26348v1
• Date: Tue, 28 Jul 2026 23:43:00 GMT
• Title: When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses
• Abstract summary: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions.<n>We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems.<n>A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data.<n>Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data.
• Score: 4.327699139300454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems. A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data: U.S. general social attitudes (General Social Survey) and cross-cultural values (World Values Survey). Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data. Under demographic prompting and the survey-simulation protocols we test, two failures replicate across both domains, all four models, and both families. First, at the individual level no LLM beats even the strongest baseline; on cross-cultural values every model falls well below it, and the gap survives distance-aware and proper scoring. Second, models systematically over-determine demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question-group combination and robust to a coding-invariant measure. Neither failure is remedied by a larger, more capable model. A decision-impact analysis shows why this matters in practice: on a segment-targeting task the models inflate between-segment gaps two to fourfold, would direct a team to the wrong segment in half of U.S. and most cross-cultural cases, and manufacture segment splits that do not exist in real people. We make the cross-domain benchmark and the evaluation framework available on request, so that teams can determine in advance when synthetic-user evidence is safe for decision support and when it is not.

Related papers

• Analyzing and Correcting Benevolence Bias in Large Language Models [25.94213078450494]
  Large language models (LLMs) are increasingly used as stand-ins for human respondents.<n>Here we identify and measure benevolence bias, a small but consistent tendency for aligned LLMs to lean toward the kinder, safer, more socially approved answer on value-laden survey questions.<n>We find that the bias is a stable model property, not a quirk of any one system.
  arXiv  Detail & Related papers  (2026-07-26T08:37:58Z)
• A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models [0.0]
  This paper introduces a consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness.<n>We conduct a controlled study using five state-of-the-art LLMs across multiple domains, including programming, general knowledge, safety, logical reasoning, and mathematics.<n>Scores are aggregated into a Relative Intelligence Index (RII), representing how frequently a model's responses are preferred by other models.
  arXiv  Detail & Related papers  (2026-07-19T08:59:19Z)
• DualEval: Joint Model-Item Calibration for Unified LLM Evaluation [67.99119926012686]
  DualEval is a latent model-item calibration framework that represents models and evaluation items in a shared space.<n>We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries.
  arXiv  Detail & Related papers  (2026-06-24T22:40:46Z)
• MANTA: Multi-turn Assessment for Nonhuman Thinking & Alignment [0.1452394725421793]
  MANTA is a dynamic multi-turn evaluation framework built on the Inspect AI platform.<n>We present preliminary results from evaluations of claude-sonnet-4-20250514 and openai/gpt-4o.<n>We additionally present STYLEJUDGE, a controlled four-judge study demonstrating systematic format bias in LLM-as-judge evaluation.
  arXiv  Detail & Related papers  (2026-04-18T19:51:32Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework [0.31497178896386996]
  HUMAINE is a framework for multidimensional, demographically aware measurement of human-AI interaction.<n>We evaluate 28 state-of-the-art models across five human-centric dimensions.
  arXiv  Detail & Related papers  (2026-02-03T09:51:47Z)
• The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality [70.45240108873001]
  The FACTS Leaderboard is an online leaderboard suite that comprehensively evaluates the ability of language models to generate factually accurate text.<n>The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards.
  arXiv  Detail & Related papers  (2025-12-11T16:35:14Z)
• Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) [90.45301024940329]
  Language models (LMs) often struggle to generate diverse, human-like creative content.<n>We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries.<n>We present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect.
  arXiv  Detail & Related papers  (2025-10-27T03:16:21Z)
• Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble [46.82793004650415]
  Large language models (LLMs) have demonstrated promise in emulating human-like responses across a range of tasks.<n>We propose a novel alignment framework that treats LLMs as agent proxies for human survey respondents.<n>We introduce P2P, a system that steers LLM agents toward representative behavioral patterns using structured prompt engineering, entropy-based sampling, and regression-based selection.
  arXiv  Detail & Related papers  (2025-09-14T15:08:45Z)
• MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Framework [53.82097200295448]
  Mean-Field LLM (MF-LLM) is first to incorporate mean field theory into social simulation.<n>MF-LLM models bidirectional interactions between individuals and the population through an iterative process.<n> IB-Tune is a novel fine-tuning method inspired by the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-04-30T12:41:51Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.