When Plausible Is Not Realistic: Evaluating Human Mobility in LLM-Based Urban Simulation

• URL: http://arxiv.org/abs/2606.13835v1
• Date: Thu, 11 Jun 2026 19:03:52 GMT
• Title: When Plausible Is Not Realistic: Evaluating Human Mobility in LLM-Based Urban Simulation
• Abstract summary: We introduce a validation framework for evaluating the mobility of generative agents of urban simulators against real-world data.<n>Our analysis reveals a substantial gap between narrative plausibility and empirical mobility realism.
• Score: 1.7205106391379026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based generative agents are increasingly used in urban simulators, yet it remains unclear whether they reproduce empirically realistic human mobility patterns or merely generate plausible mobility narratives. We introduce a validation framework for evaluating the mobility of generative agents of LLM-based urban simulators against real-world mobility data. For this, we use mobility laws, temporal rhythms, network motifs, semantic activity transitions, and behavioral mobility profiles. Using datasets from the Greater Paris region and Shanghai, we evaluate AgentSociety and CitySim across multiple dimensions of mobility realism. Our analysis reveals a substantial gap between narrative plausibility and empirical mobility realism. Although the simulators capture some high-level semantic activity distributions, they struggle to reproduce core spatial and temporal constraints, including realistic trip-length distributions, origin-destination flows, dwell times, and transition dynamics. We further observe that realistic mobility diversity is unstable across default prompting configurations and may require explicit profile-aware initialization. To support reproducible evaluation, we also contribute scalable and open LLM-driven infrastructure for regional-scale map generation, observability-enhanced simulation, mobility-metric computation, and traffic simulation. Our findings highlight the need for rigorous empirical validation of LLM-based urban simulators and provide practical tools for building more realistic and reproducible urban simulation systems.

Related papers

• CityBehavEx: A Scalable and Empirically Validated LLM-Assisted Urban Simulation Platform [1.7205106391379026]
  We present CityBehavEx, an interactive LLM-assisted urban simulation platform that scales to city-size populations.<n>Instead of invoking large language models for every agent action, CityBehavEx combines established human mobility models with fine-tuned cross-encoders.<n>This design enables large-scale simulations, as demonstrated in a case study of 100,000 agents over 75 days in under one hour on a single consumer GPU.
  arXiv  Detail & Related papers  (2026-07-13T19:03:25Z)
• VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation [61.82502719679122]
  We introduce VLNVerse, a benchmark for Versatile, Embodied, Realistic Simulation, and Evaluation.<n>VLNVerse redefines VLN as a scalable, full-stack embodied AI problem.<n>We propose a novel unified multi-task model capable of addressing all tasks within the benchmark.
  arXiv  Detail & Related papers  (2025-12-22T04:27:26Z)
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
  We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
  arXiv  Detail & Related papers  (2025-10-20T13:14:38Z)
• Incorporating LLMs for Large-Scale Urban Complex Mobility Simulation [0.04517077427559345]
  This study presents an innovative approach to urban mobility simulation by integrating a Large Language Model (LLM) with Agent-Based Modeling (ABM)<n>Unlike traditional rule-based ABM, the proposed framework leverages LLM to enhance agent diversity and realism.<n>Using real-world data, the simulation models individual behaviors and large-scale mobility patterns in Taipei City.
  arXiv  Detail & Related papers  (2025-05-28T01:54:28Z)
• MobileCity: An Efficient Framework for Large-Scale Urban Behavior Simulation [24.339278159507035]
  We present MobileCity, a lightweight simulation platform designed to model realistic urban mobility with high computational efficiency.<n>We introduce a comprehensive transportation system with multiple transport modes, and collect questionnaire data from respondents to construct agent profiles.<n>To enable scalable simulation, agents perform action selection within a pre-generated action space and uses local models for efficient agent memory generation.
  arXiv  Detail & Related papers  (2025-04-18T07:01:05Z)
• TrajLLM: A Modular LLM-Enhanced Agent-Based Framework for Realistic Human Trajectory Simulation [3.8106509573548286]
  This work leverages Large Language Models (LLMs) to simulate human mobility, addressing challenges like high costs and privacy concerns in traditional models.<n>Our hierarchical framework integrates persona generation, activity selection, and destination prediction, using real-world demographic and psychological data.
  arXiv  Detail & Related papers  (2025-02-26T00:13:26Z)
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation [53.85002640149283]
  Key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge. This study identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models.
  arXiv  Detail & Related papers  (2023-09-01T19:29:53Z)
• TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction [149.5716746789134]
  We show data-driven traffic simulation can be formulated as a world model. We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving. Experiments on the open motion dataset show TrafficBots can simulate realistic multi-agent behaviors.
  arXiv  Detail & Related papers  (2023-03-07T18:28:41Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.