GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

• URL: http://arxiv.org/abs/2608.07411v1
• Date: Fri, 07 Aug 2026 16:58:04 GMT
• Title: GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks
• Abstract summary: We present benchName, a benchmark for probing Large Language Models on geo-related tasks.<n>We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains.<n>Our results show that reasoning and size have a strong impact on overall performance.
• Score: 1.5249435285717097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. In this paper, we present \benchName, a comprehensive benchmark for probing LLMs on geo-related tasks. We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Our results show that reasoning and size have a strong impact on overall performance. GeoBenchLLM is publicly available at https://github.com/Rfr2003/GeoBenchLLM.

Related papers

• EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration [3.059053384364342]
  We introduce EVGeoQA, a novel benchmark built upon Electric Vehicle (EV) charging scenarios.<n>Each query in EVGeoQA is explicitly bound to a user's real-time coordinate and integrates the dual objectives of a charging necessity and a co-located activity preference.<n>Our experiments reveal that while LLMs successfully utilize tools to address sub-tasks, they struggle with long-range spatial exploration.
  arXiv  Detail & Related papers  (2026-04-08T13:20:38Z)
• GeoX-Bench: Benchmarking Cross-View Geo-Localization and Pose Estimation Capabilities of Large Multimodal Models [78.98542840563907]
  GeoX-Bench is a comprehensive underlineBenchmark designed to explore and evaluate the capabilities of LMMs.<n>It contains 10,859 panoramic-satellite image pairs spanning 128 cities in 49 countries, along with corresponding 755,976 question-answering (QA) pairs.<n>Based on GeoX-Bench, we evaluate the capabilities of 25 state-of-the-art LMMs on cross-view geo-localization and pose estimation tasks.
  arXiv  Detail & Related papers  (2025-11-17T11:19:07Z)
• Coordinates from Context: Using LLMs to Ground Complex Location References [1.016973253129328]
  Geocoding is the task of linking a location reference to an actual geographic location.<n>We show that a relatively small fine-tuned LLM can achieve comparable performance with much larger off-the-shelf models.
  arXiv  Detail & Related papers  (2025-10-09T18:51:52Z)
• OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence [51.0456395687016]
  multimodal large language models (LLMs) have opened new frontiers in artificial intelligence.<n>We propose a MLLM (OmniGeo) tailored to geospatial applications.<n>By combining the strengths of natural language understanding and spatial reasoning, our model enhances the ability of instruction following and the accuracy of GeoAI systems.
  arXiv  Detail & Related papers  (2025-03-20T16:45:48Z)
• GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks [84.86699025256705]
  We present GEOBench-VLM, a benchmark specifically designed to evaluate Vision-Language Models (VLMs) on geospatial tasks.<n>Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales.<n>We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges.
  arXiv  Detail & Related papers  (2024-11-28T18:59:56Z)
• An LLM Agent for Automatic Geospatial Data Analysis [5.842462214442362]
  Large language models (LLMs) are being used in data science code generation tasks. Their application to geospatial data processing is challenging due to difficulties in incorporating complex data structures and spatial constraints. We introduce GeoAgent, a new interactive framework designed to help LLMs handle geospatial data processing more effectively.
  arXiv  Detail & Related papers  (2024-10-24T14:47:25Z)
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models [49.20315582673223]
  We present GeoLLM, a novel method that can effectively extract geospatial knowledge from large language models. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe.
  arXiv  Detail & Related papers  (2023-10-10T00:03:23Z)
• K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization [105.89544876731942]
  Large language models (LLMs) have achieved great success in general domains of natural language processing. We present the first-ever LLM in geoscience, K2, alongside a suite of resources developed to further promote LLM research within geoscience.
  arXiv  Detail & Related papers  (2023-06-08T09:29:05Z)
• MGeo: Multi-Modal Geographic Pre-Training Method [49.78466122982627]
  We propose a novel query-POI matching method Multi-modal Geographic language model (MGeo) MGeo represents GC as a new modality and is able to fully extract multi-modal correlations for accurate query-POI matching. Our proposed multi-modal pre-training method can significantly improve the query-POI matching capability of generic PTMs.
  arXiv  Detail & Related papers  (2023-01-11T03:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.