UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

• URL: http://arxiv.org/abs/2608.27456v1
• Date: Thu, 27 Aug 2026 17:59:33 GMT
• Title: UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City
• Abstract summary: Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move.<n>We investigate how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city.<n>We propose UrbanGround, the first sandbox to make this question testable in a physically constrained replica of Hong Kong built from territory-wide 3D geospatial data.
• Score: 63.28750191463138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move. In this paper, we investigate how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city. We propose UrbanGround, the first sandbox to make this question testable in a physically constrained replica of Hong Kong built from territory-wide 3D geospatial data. UrbanGround supports closed-loop interaction from a first-person view and provides an interactive map for navigation. Agents can directly enter the 3D city and explore from a first-person view. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an agent can ground a local scene well enough to answer spatial questions after active observation. Then we ask whether that grounding supports navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. Contemporary MLLM agents usually show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far current MLLM agents can explore reliably in complex, open-ended urban environments.

Related papers

• How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace [27.624616950756643]
  Large multimodal models (LMMs) show strong visual-linguistic reasoning but their capacity for spatial decision-making and action remains unclear.<n>We investigate whether LMMs can achieve embodied spatial action like human through a challenging scenario: goal-oriented navigation in urban 3D spaces.<n>Experiments show that current LMMs exhibit emerging action capabilities, yet remain far from human-level performance.
  arXiv  Detail & Related papers  (2026-04-09T08:37:20Z)
• City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs [13.863236619171174]
  Task is designed to evaluate sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environments.<n>We operationalize this task with CityNav, a benchmark encompassing four diverse global cities.<n>Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points.<n>We propose Verbalization of Path (VoP), which explicitly grounds the agent's internal reasoning by probing an explicit cognitive map.
  arXiv  Detail & Related papers  (2025-12-17T19:59:31Z)
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks [108.15756345836901]
  We provide a comprehensive review of multimodal spatial reasoning tasks with large models.<n>We review advances in embodied AI, including vision-language navigation and action models.<n>We consider emerging modalities such as audio and egocentric video, which contribute to novel spatial understanding through new sensors.
  arXiv  Detail & Related papers  (2025-10-29T17:55:43Z)
• OpenCity3D: What do Vision-Language Models know about Urban Environments? [4.468922393753096]
  Vision-language models (VLMs) show great promise for 3D scene understanding but are mainly applied to indoor spaces or autonomous driving.<n>This work expands their use to urban-scale environments by leveraging 3D reconstructions from multi-view aerial imagery.<n>We propose OpenCity3D, an approach that addresses high-level tasks, such as population density estimation, building age classification, property price prediction, crime rate assessment, and noise pollution evaluation.
  arXiv  Detail & Related papers  (2025-03-21T01:11:21Z)
• CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space [35.223263448229716]
  Embodied Question Answering (EQA) has primarily focused on indoor environments.<n>We introduce CityEQA, a new task where an embodied agent answers open-vocabulary questions through active exploration in dynamic city spaces.<n>We present CityEQA-EC, the first benchmark dataset featuring 1,412 human-annotated tasks.<n>We also propose Planner-Manager-Actor (PMA), a novel agent tailored for CityEQA.
  arXiv  Detail & Related papers  (2025-02-18T04:36:15Z)
• CityX: Controllable Procedural Content Generation for Unbounded 3D Cities [50.10101235281943]
  Current generative methods fall short in either diversity, controllability, or fidelity.<n>In this work, we resort to the procedural content generation (PCG) technique for high-fidelity generation.<n>We develop a multi-agent framework to transform multi-modal instructions, including OSM, semantic maps, and satellite images, into executable programs.<n>Our method, named CityX, demonstrates its superiority in creating diverse, controllable, and realistic 3D urban scenes.
  arXiv  Detail & Related papers  (2024-07-24T18:05:13Z)
• 3D Question Answering for City Scene Understanding [12.433903847890322]
  3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. We introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA.
  arXiv  Detail & Related papers  (2024-07-24T16:22:27Z)
• ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints [94.60414567852536]
  Long-range navigation requires both planning and reasoning about local traversability. We propose a learning-based approach that integrates learning and planning. ViKiNG can leverage its image-based learned controller and goal-directed to navigate to goals up to 3 kilometers away.
  arXiv  Detail & Related papers  (2022-02-23T02:14:23Z)
• Occupancy Anticipation for Efficient Exploration and Navigation [97.17517060585875]
  We propose occupancy anticipation, where the agent uses its egocentric RGB-D observations to infer the occupancy state beyond the visible regions. By exploiting context in both the egocentric views and top-down maps our model successfully anticipates a broader map of the environment. Our approach is the winning entry in the 2020 Habitat PointNav Challenge.
  arXiv  Detail & Related papers  (2020-08-21T03:16:51Z)
• Learning to Move with Affordance Maps [57.198806691838364]
  The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent. Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry. We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.
  arXiv  Detail & Related papers  (2020-01-08T04:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.