Can Large Language Models Represent Urban Publics? Behavioral Replication and Population Mismatch in an Affordable-Housing Experiment

• URL: http://arxiv.org/abs/2607.27100v1
• Date: Wed, 29 Jul 2026 16:27:36 GMT
• Title: Can Large Language Models Represent Urban Publics? Behavioral Replication and Population Mismatch in an Affordable-Housing Experiment
• Abstract summary: We show that large language models can approximate one aggregate contrast while failing to preserve the population structure.<n>Model evaluation in urban planning should test whether this spatial and social structure survives simulation, not only average effects.
• Score: 1.7984459258809926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There is growing interest in using large language models (LLMs) as low-cost proxies for resident attitudes in urban planning. Previous work shows that LLMs can predict average results of survey experiments, but less is known about whether they preserve the spatially anchored, identity-conditioned structure behind those averages, namely how support changes as a project approaches homes and how that response divides across tenure and partisan groups. We compared eight open-weight LLMs with 843 respondents in a US affordable-housing survey experiment, testing whether they reproduced the owner-renter difference in support change as a proposed development moved from 2 miles to 1/8 mile. Qwen 2.5 14B was closest (-0.242 versus the human -0.285) and was the only model to meet the prespecified +/-0.20 equivalence criterion; Phi-4 14B was directionally aligned but attenuated (-0.150), and other models showed weak, null, or reversed moderation. This aggregate match masked structural failure. Qwen attenuated the Republican contrast and exaggerated the Independent one, its RMSE across 27 party-by-tenure-by-item cells was 0.613, its median model-to-human variance ratio was 0.099, and question order shifted the contrast by +0.367. Identity-cue removal and selective nonresponse changed which comparisons were estimable, and rationale-first responses differed from matched direct-choice responses in 20.6-35.3% of focal comparisons. An LLM can thus approximate one aggregate contrast while failing to preserve the population structure, within-group heterogeneity, and measurement stability that generate it. Model evaluation in urban planning should test whether this spatial and social structure survives simulation, not only average effects.

Related papers

• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• Small Foundation Models of Human Cognition and Behaviour [45.88028371034407]
  We train 14 models from 135M to 14B parameters across four architecture families on Psych-101.<n>Psych-101 is a dataset of 10.7 million trial-level choices from 160 experiments.<n>We show that small cognitively fine-tuned models show promise as noise ceiling estimators for psychological experiments.
  arXiv  Detail & Related papers  (2026-08-05T11:34:20Z)
• Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel [0.0]
  We study a fixed panel of sixteen lightweight persona-conditioned GPT-4.1 configurations in repeated strategic games.<n>Variation was strongly prompt-indexed, but its share depended on uncertainty assumptions.<n>Results concern one fixed model-prompt panel and do not establish human substitutability.
  arXiv  Detail & Related papers  (2026-08-02T04:10:11Z)
• Analyzing and Correcting Benevolence Bias in Large Language Models [25.94213078450494]
  Large language models (LLMs) are increasingly used as stand-ins for human respondents.<n>Here we identify and measure benevolence bias, a small but consistent tendency for aligned LLMs to lean toward the kinder, safer, more socially approved answer on value-laden survey questions.<n>We find that the bias is a stable model property, not a quirk of any one system.
  arXiv  Detail & Related papers  (2026-07-26T08:37:58Z)
• Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions [4.691907035988758]
  Large language model (LLM) agents offer a generative approach to simulating human behavior under conditions that may have few or no direct historical analogues.<n>We evaluate whether empirical grounding improves the statistical realism of LLM-agent simulations during disruptions.
  arXiv  Detail & Related papers  (2026-07-19T23:06:28Z)
• Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling [0.0]
  Synthetic-population tools run every individual as an independent large language model (LLM) agent.<n>We show that this paradigm has a basic failure mode, and we set a distribution-first corrective against it.
  arXiv  Detail & Related papers  (2026-07-17T15:45:57Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models [45.42736932931535]
  Law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population.<n>We use binary trees as an evaluation scaffold to partition a population into increasingly fine-grained subpopulations.<n>We prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity.
  arXiv  Detail & Related papers  (2026-07-16T17:59:31Z)
• Silicon Sampling via Cross-Survey Transfer [0.0]
  Silicon sampling-using large language models (LLMs) to simulate human survey respondents has emerged as a promising approach for augmenting traditional survey research.<n>We propose a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey.
  arXiv  Detail & Related papers  (2026-07-03T08:22:18Z)
• SL-BiLEM: Structured Learnable Behavior-in-the-Loop Epidemic Modeling for Forecasting and Policy Evaluation [42.644852156101685]
  Epidemic forecasting faces a fundamental challenge: human behavior dynamically responds to disease spread.<n>We propose textbfSL-BiLEM (Structured Learnable Behavior-in-the-Loop Epidemic Model), leveraging physical constraints as regularization for robust extrapolation.
  arXiv  Detail & Related papers  (2026-05-26T08:46:04Z)
• Benchmarking LLMs for Community Governance Simulation with Life-history Narratives [46.86050402684712]
  Large language models (LLMs) are scalable to simulate human attitudes and behaviors at low cost.<n>We present an integrated research framework covering dataset, benchmark, algorithm, and system.<n>System integrates curriculum-LoRA into a closed-loop policy-evaluation pipeline.
  arXiv  Detail & Related papers  (2026-05-22T15:48:49Z)
• Preserving Disagreement: Architectural Heterogeneity and Coherence Validation in Multi-Agent Policy Simulation [0.0]
  Multi-agent deliberation systems using large language models (LLMs) are increasingly proposed for policy simulation.<n>They suffer from artificial consensus: evaluator agents converge on the same option regardless of their assigned value perspectives.<n>We present the AI Council, a three-phase deliberation framework, and conduct 120 deliberations across two policy scenarios to test two interventions.
  arXiv  Detail & Related papers  (2026-04-29T11:47:28Z)
• PsychBench: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations [0.0]
  Large language models are increasingly deployed to simulate patients for clinical training, research, and mental health tools.<n>We introduce PsychBench, the first epidemiological audit of LLM patient simulation.<n>We show that models produce clinically plausible individuals while misrepresenting the populations they are drawn from.
  arXiv  Detail & Related papers  (2026-04-19T10:05:25Z)
• Evaluating Perceptual Distance Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data [43.714290271351466]
  This paper introduces a more robust distance-model evaluation method using a pure probabilistic approach, applying maximum likelihood estimation to a binomial decision model.<n>Our method demonstrates superior simplicity, interpretability, flexibility, and computational efficiency, as shown through evaluations of various visual distance models on two 2AFC PF datasets.
  arXiv  Detail & Related papers  (2024-03-15T15:21:04Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.