The average-farmer illusion in language-model simulations of agricultural decisions

• URL: http://arxiv.org/abs/2609.15038v1
• Date: Mon, 14 Sep 2026 04:54:51 GMT
• Title: The average-farmer illusion in language-model simulations of agricultural decisions
• Abstract summary: Language-model agents are increasingly used as synthetic people in surveys and social simulations.<n>We compare Claude, Codex and Kimi under four prespecified prompt designs with matched farmer decisions from China and four African countries.
• Score: 9.14716652210035
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Language-model agents are increasingly used as synthetic people in surveys and social simulations, yet their apparent realism is often judged from population averages or distributional similarity. We tested what such evidence actually establishes by comparing Claude, Codex and Kimi under four prespecified prompt designs with matched farmer decisions from China and four African countries. Some configurations reproduced observed means and adoption rates. However, their person-level predictions were weak; their decisions clustered around typical values and policy-relevant extremes were largely missing. Most strikingly, a simple generator fitted only to the observed marginal dis- tribution, and given no information about any farmer, achieved greater distributional similarity than every language-model configuration. Prompt additions produced conditional gains rather than uni- versal improvement: results varied with model, outcome, population and validation target. We call this the average-farmer illusion: a synthetic population can look realistic while failing to repro- duce who does what or how behaviour varies. We provide a claim-matched validation framework and reusable modular prompts that turn prompt construction into an auditable experimental process. Population-level resemblance should therefore be treated as the start of validation, not as evidence of individual simulation.

Related papers

• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation [7.947622351777421]
  We show that some studies find promising alignment with human survey data, while others find persona collapse and weak demographic sensitivity.<n>We call the first task emulation, in which models generate individual responses that aggregate into a population distribution.<n>We call the second task estimation, in which models directly predict the population distribution.
  arXiv  Detail & Related papers  (2026-08-04T02:51:38Z)
• Improving Cross-Cultural Survey Simulation with Calibrated Value Personas [2.5964189305343712]
  Large language models (LLMs) are increasingly used to simulate human opinions and survey responses, but their ability to reproduce population responses across cultures remains limited.<n>We propose a value-based persona construction method that derives textual descriptors from survey responses capturing core cultural dimensions.<n>We show that our approach reduces prediction error across countries, with the largest improvements observed in underrepresented populations.
  arXiv  Detail & Related papers  (2026-05-15T17:10:50Z)
• How Well Do Large Language Models Capture Human Personality? [38.97721735513693]
  Large language models (LLMs) are increasingly used to simulate human populations via persona prompting.<n>In this work, we formalize assumptions that richer persona descriptions improve behavioral fidelity, similarly sized attribute combinations are equally simulatable, and persona definitions generalize across tasks.<n>We identify a fundamental limitation we term persona manifold collapse, where increasingly expressive persona specifications lead to systematic contraction of representational and behavioral diversity.
  arXiv  Detail & Related papers  (2026-05-12T10:33:48Z)
• The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice [22.228743542695835]
  We study a three-source design that combines a large confounded observational log for scale, a small randomized experiment for unconfounded scoring, and an offline simulator.<n>Our main result is an identification theorem showing that the randomized experiment and the simulator are together enough to recover causal model values.
  arXiv  Detail & Related papers  (2026-05-02T07:55:55Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• Evaluation of Large Language Models via Coupled Token Generation [19.187846871216568]
  State of the art large language models rely on randomization to respond to a prompt.<n>We argue that the evaluation and ranking of large language models should control for the randomization underpinning their functioning.
  arXiv  Detail & Related papers  (2025-02-03T19:01:17Z)
• Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? [90.35044668396591]
  A recurring theme in machine learning is algorithmic monoculture: the same systems, or systems that share components, are deployed by multiple decision-makers. We propose the component-sharing hypothesis: if decision-makers share components like training data or specific models, then they will produce more homogeneous outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that sharing training data reliably exacerbates homogenization. We conclude with philosophical analyses of and societal challenges for outcome homogenization, with an eye towards implications for deployed machine learning systems.
  arXiv  Detail & Related papers  (2022-11-25T09:33:11Z)
• Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
  Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
  arXiv  Detail & Related papers  (2022-01-27T22:15:56Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Plinko: A Theory-Free Behavioral Measure of Priors for Statistical Learning and Mental Model Updating [62.997667081978825]
  We present three experiments using "Plinko", a behavioral task in which participants estimate distributions of ball drops over all available outcomes. We show that participant priors cluster around prototypical probability distributions and that prior cluster membership may indicate learning ability. We verify that individual participant priors are reliable representations and that learning is not impeded when faced with a physically implausible ball drop distribution.
  arXiv  Detail & Related papers  (2021-07-23T22:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.