CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities

• URL: http://arxiv.org/abs/2605.26036v1
• Date: Mon, 25 May 2026 17:03:46 GMT
• Title: CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities
• Abstract summary: CityRep is a benchmark that evaluates urban representations across data modalities, cities, and tasks using spatially structured splits.<n>Results show that performance is highly sensitive to the split protocol, with random splits inflating scores and altering model rankings.<n>CityRep is released as a reproducible benchmark with datasets, evaluation pipelines, and diagnostic tools to facilitate fair comparison.
• Score: 0.8319452826580411
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Urban representation learning encodes complex urban environments into general-purpose embeddings for diverse downstream tasks and emerging urban foundation models. However, current evaluations are limited, typically focusing on one or two cities and tasks and relying on random splits that introduce spatial leakage, leading to inflated performance and weak support for cross-location generalization and fair comparison. To address this, we propose CityRep, a unified benchmark that evaluates urban representations across data modalities, cities, and tasks using spatially structured splits. CityRep consists of three key components: (1) a spatial unit-agnostic evaluation framework that supports heterogeneous urban representations through a standardized alignment module; (2) a unified evaluation protocol using block-based spatial splits to mitigate spatial leakage and enable rigorous model comparison; and (3) an extensible multi-city, multi-task benchmark suite spanning 8 cities and 8 tasks across regression, classification, and distribution prediction. We evaluate 11 representative urban representation models. Results show that performance is highly sensitive to the split protocol, with random splits inflating scores and altering model rankings. We also observe substantial variability across cities and tasks, underscoring the need for generalization-aware evaluation. CityRep is released as a reproducible benchmark with datasets, evaluation pipelines, and diagnostic tools to facilitate fair comparison and support future research in urban representation learning towards urban foundation models.

Related papers

• An Empirical Evaluation of Cross-City POI Recommendation on a Large-Scale Benchmark [55.73054722370559]
  Cross-city point-of-interest (POI) recommendation is crucial for navigating unfamiliar urban environments.<n>We re-examine whether conclusions drawn on small prior benchmarks still hold under worldwide coverage, low home-destination region overlap, and large, semantically rich POI inventories.
  arXiv  Detail & Related papers  (2026-08-28T02:16:33Z)
• Mapping the City Through the Lens of Language Models [5.705725225868815]
  Ten open-weight checkpoints rate anonymized profiles derived from real morphological urban centres.<n>The framework makes an otherwise vague notion of what models regard as an ordinary city empirically traceable.
  arXiv  Detail & Related papers  (2026-08-04T00:09:08Z)
• Multi-Agent Collaborative Reasoning with Tool-Augmented Evidence for Urban Region Profiling [31.71830721319945]
  Urban region profiling supports applications such as population estimation, economic assessment, and environmental monitoring.<n>We propose UrbanAgent, an agentic framework that reframes urban region profiling as a reasoning-driven inference problem.<n>Experiments on global urban datasets for Carbon emissions, GDP, and Population estimation show that UrbanAgent consistently outperforms existing baselines.
  arXiv  Detail & Related papers  (2026-07-15T08:01:42Z)
• UrbanVerse: Learning Urban Region Representation Across Cities and Tasks [18.711357897379283]
  UrbanVerse is a model for cross-city urban representation learning and cross-task urban analytics.<n>For cross-city generalization, UrbanVerse focuses on features local to the target regions and structural features of the nearby regions rather than the entire city.<n>Experiments on real-world datasets show that UrbanVerse consistently outperforms state-of-the-art methods across six tasks under cross-city settings.
  arXiv  Detail & Related papers  (2026-02-17T17:28:48Z)
• UrbanMoE: A Sparse Multi-Modal Mixture-of-Experts Framework for Multi-Task Urban Region Profiling [47.568568425459716]
  We develop a benchmark for multi-task urban region profiling, featuring multi-modal features and a diverse set of strong baselines.<n>We then propose UrbanMoE, the first sparse multi-modal, multi-expert framework specifically architected to solve the multi-task challenge.<n>We conduct extensive experiments on three real-world datasets within our benchmark, where UrbanMoE consistently demonstrates superior performance over all baselines.
  arXiv  Detail & Related papers  (2026-01-30T09:25:05Z)
• Urban-R1: Reinforced MLLMs Mitigate Geospatial Biases for Urban General Intelligence [64.36291202666212]
  Urban General Intelligence (UGI) refers to AI systems that can understand and reason about complex urban environments.<n>Recent studies have built urban foundation models using supervised fine-tuning (SFT) of LLMs and MLLMs.<n>We propose Urban-R1, a reinforcement learning-based post-training framework that aligns MLLMs with the objectives of UGI.
  arXiv  Detail & Related papers  (2025-10-18T15:59:09Z)
• Collaborative Imputation of Urban Time Series through Cross-city Meta-learning [54.438991949772145]
  We propose a novel collaborative imputation paradigm leveraging meta-learned implicit neural representations (INRs)<n>We then introduce a cross-city collaborative learning scheme through model-agnostic meta learning.<n>Experiments on a diverse urban dataset from 20 global cities demonstrate our model's superior imputation performance and generalizability.
  arXiv  Detail & Related papers  (2025-01-20T07:12:40Z)
• UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios [60.492736455572015]
  We present UrBench, a benchmark designed for evaluating LMMs in complex multi-view urban scenarios.<n>UrBench contains 11.6K meticulously curated questions at both region-level and role-level.<n>Our evaluations on 21 LMMs show that current LMMs struggle in the urban environments in several aspects.
  arXiv  Detail & Related papers  (2024-08-30T13:13:35Z)
• Explainable Hierarchical Urban Representation Learning for Commuting Flow Prediction [1.5156879440024378]
  Commuting flow prediction is an essential task for municipal operations in the real world. We develop a heterogeneous graph-based model to generate meaningful region embeddings for predicting different types of inter-level OD flows. Our proposed model outperforms existing models in terms of a uniform urban structure.
  arXiv  Detail & Related papers  (2024-08-27T03:30:01Z)
• Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks [82.82866901799565]
  We build a new set of multimodal remote sensing benchmark datasets (including hyperspectral, multispectral, SAR) for the study purpose of the cross-city semantic segmentation task. Beyond the single city, we propose a high-resolution domain adaptation network, HighDAN, to promote the AI model's generalization ability from the multi-city environments. HighDAN is capable of retaining the spatially topological structure of the studied urban scene well in a parallel high-to-low resolution fusion fashion.
  arXiv  Detail & Related papers  (2023-09-26T23:55:39Z)
• Methodological Foundation of a Numerical Taxonomy of Urban Form [62.997667081978825]
  We present a method for numerical taxonomy of urban form derived from biological systematics. We derive homogeneous urban tissue types and, by determining overall morphological similarity between them, generate a hierarchical classification of urban form. After framing and presenting the method, we test it on two cities - Prague and Amsterdam.
  arXiv  Detail & Related papers  (2021-04-30T12:47:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.