OBSR: Open Benchmark for Spatial Representations

• URL: http://arxiv.org/abs/2510.05879v2
• Date: Thu, 09 Oct 2025 10:19:28 GMT
• Title: OBSR: Open Benchmark for Spatial Representations
• Authors: Julia Moska, Oleksii Furman, Kacper Kozaczko, Szymon Leszkiewicz, Jakub Polczyk, Piotr Gramacki, Piotr Szymański,
• Abstract summary: This paper introduces a novel benchmark designed to assess the performance, accuracy, and efficiency of geospatial embedders.<n>Our benchmark is modality-agnostic and comprises 7 distinct datasets from diverse cities across three continents.
• Score: 0.3936827689390718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GeoAI is evolving rapidly, fueled by diverse geospatial datasets like traffic patterns, environmental data, and crowdsourced OpenStreetMap (OSM) information. While sophisticated AI models are being developed, existing benchmarks are often concentrated on single tasks and restricted to a single modality. As such, progress in GeoAI is limited by the lack of a standardized, multi-task, modality-agnostic benchmark for their systematic evaluation. This paper introduces a novel benchmark designed to assess the performance, accuracy, and efficiency of geospatial embedders. Our benchmark is modality-agnostic and comprises 7 distinct datasets from diverse cities across three continents, ensuring generalizability and mitigating demographic biases. It allows for the evaluation of GeoAI embedders on various phenomena that exhibit underlying geographic processes. Furthermore, we establish a simple and intuitive task-oriented model baselines, providing a crucial reference point for comparing more complex solutions.

Related papers

• GeoAgent: Learning to Geolocate Everywhere with Reinforced Geographic Characteristics [91.17301794848025]
  This paper presents GeoAgent, a model capable of reasoning closely with humans and deriving fine-grained address conclusions.<n>Previous RL-based methods have achieved breakthroughs in performance and interpretability but still remain concerns because of their reliance on AI-generated chain-of-thought (CoT) data and training strategies.
  arXiv  Detail & Related papers  (2026-02-13T04:48:05Z)
• GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI [52.13138825802668]
  GeoFMs are transforming Earth Observation, but evaluation lacks standardized protocols.<n> GEO-Bench-2 addresses this with a comprehensive framework spanning classification, segmentation, regression, object detection, and instance segmentation.<n>Code, data, and leaderboard for GEO-Bench-2 are publicly released under a permissive license.
  arXiv  Detail & Related papers  (2025-11-19T17:45:02Z)
• GeoBS: Information-Theoretic Quantification of Geographic Bias in AI Models [34.611626290720295]
  We establish an information-theoretic framework for geo-bias evaluation, called GeoBS (Geo-Bias Scores)<n>We propose three novel geo-bias scores that explicitly take intricate spatial factors into consideration.
  arXiv  Detail & Related papers  (2025-09-27T20:07:21Z)
• G-OSR: A Comprehensive Benchmark for Graph Open-Set Recognition [54.45837774534411]
  We introduce textbfG-OSR, a benchmark for evaluating Graph Open-Set Recognition (GOSR) methods at both the node and graph levels.<n>Results offer critical insights into the generalizability and limitations of current GOSR methods.
  arXiv  Detail & Related papers  (2025-03-01T13:02:47Z)
• GeoJEPA: Towards Eliminating Augmentation- and Sampling Bias in Multimodal Geospatial Learning [0.0]
  We present GeoJEPA, a versatile multimodal fusion model for geospatial data built on the self-supervised Joint-Embedding Predictive Architecture.<n>We aim to eliminate the widely accepted augmentation- and sampling biases found in self-supervised geospatial representation learning.<n>The results are multimodal semantic representations of urban regions and map entities that we evaluate both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2025-02-25T22:03:28Z)
• Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework [59.42946541163632]
  We introduce a comprehensive geolocation framework with three key components.<n>GeoComp, a large-scale dataset; GeoCoT, a novel reasoning method; and GeoEval, an evaluation metric.<n>We demonstrate that GeoCoT significantly boosts geolocation accuracy by up to 25% while enhancing interpretability.
  arXiv  Detail & Related papers  (2025-02-19T14:21:25Z)
• GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks [84.86699025256705]
  We present GEOBench-VLM, a benchmark specifically designed to evaluate Vision-Language Models (VLMs) on geospatial tasks.<n>Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales.<n>We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges.
  arXiv  Detail & Related papers  (2024-11-28T18:59:56Z)
• Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework [51.26566634946208]
  We introduce smileGeo, a novel visual geo-localization framework. By inter-agent communication, smileGeo integrates the inherent knowledge of these agents with additional retrieved information. Results show that our approach significantly outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-21T03:31:30Z)
• TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning [36.725822223732635]
  We propose TorchSpatial, a learning framework and benchmark for location (point) encoding.<n>TorchSpatial contains three key components: 1) a unified location encoding framework that consolidates 15 commonly recognized location encoders; 2) the LocBench benchmark tasks encompassing 7 geo-aware image classification and 10 geo-aware image regression datasets; and 3) a comprehensive suite of evaluation metrics to quantify geo-aware model's overall performance as well as their geographic bias, with a novel Geo-Bias Score metric.
  arXiv  Detail & Related papers  (2024-06-21T21:33:16Z)
• On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence [39.86997089245117]
  Foundations models (FMs) can be adapted to a wide range of downstream tasks by fine-tuning, few-shot, or zero-shot learning. We propose that one of the major challenges of developing a FM for GeoAI is to address the multimodality nature of geospatial tasks.
  arXiv  Detail & Related papers  (2023-04-13T19:50:17Z)
• A General Purpose Neural Architecture for Geospatial Systems [142.43454584836812]
  We present a roadmap towards the construction of a general-purpose neural architecture (GPNA) with a geospatial inductive bias. We envision how such a model may facilitate cooperation between members of the community.
  arXiv  Detail & Related papers  (2022-11-04T09:58:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.