The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features

• URL: http://arxiv.org/abs/2506.00203v1
• Date: Fri, 30 May 2025 20:14:17 GMT
• Title: The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features
• Authors: Omid Reza Abbasi, Franz Welscher, Georg Weinberger, Johannes Scholz,
• Abstract summary: This study evaluates the performance of GPT-4o and Gemini 2.0 Flash in three key geospatial tasks: geocoding, elevation estimation, and reverse geocoding.<n>Neither model achieved an accurate reconstruction of Austria's federal states, highlighting persistent misclassifications.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As large language models (LLMs) continue to evolve, questions about their trustworthiness in delivering factual information have become increasingly important. This concern also applies to their ability to accurately represent the geographic world. With recent advancements in this field, it is relevant to consider whether and to what extent LLMs' representations of the geographical world can be trusted. This study evaluates the performance of GPT-4o and Gemini 2.0 Flash in three key geospatial tasks: geocoding, elevation estimation, and reverse geocoding. In the geocoding task, both models exhibited systematic and random errors in estimating the coordinates of St. Anne's Column in Innsbruck, Austria, with GPT-4o showing greater deviations and Gemini 2.0 Flash demonstrating more precision but a significant systematic offset. For elevation estimation, both models tended to underestimate elevations across Austria, though they captured overall topographical trends, and Gemini 2.0 Flash performed better in eastern regions. The reverse geocoding task, which involved identifying Austrian federal states from coordinates, revealed that Gemini 2.0 Flash outperformed GPT-4o in overall accuracy and F1-scores, demonstrating better consistency across regions. Despite these findings, neither model achieved an accurate reconstruction of Austria's federal states, highlighting persistent misclassifications. The study concludes that while LLMs can approximate geographic information, their accuracy and reliability are inconsistent, underscoring the need for fine-tuning with geographical information to enhance their utility in GIScience and Geoinformatics.

Related papers

• Performance and Generalizability Impacts of Incorporating Geolocation into Deep Learning for Dynamic PM2.5 Estimation [1.278093617645299]
  We build on a recently published deep learning-based PM2.5 estimation model that achieves state-of-the-art performance on data observed in the contiguous United States.<n>We examine three approaches for incorporating geolocation: excluding geolocation as a baseline, using raw geographic coordinates, and leveraging pretrained location encoders.
  arXiv  Detail & Related papers  (2025-05-24T02:00:34Z)
• EarthMapper: Visual Autoregressive Models for Controllable Bidirectional Satellite-Map Translation [50.433911327489554]
  We introduce EarthMapper, a novel framework for controllable satellite-map translation.<n>We also contribute CNSatMap, a large-scale dataset comprising 302,132 precisely aligned satellite-map pairs across 38 Chinese cities.<n> experiments on CNSatMap and the New York dataset demonstrate EarthMapper's superior performance.
  arXiv  Detail & Related papers  (2025-04-28T02:41:12Z)
• Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework [59.42946541163632]
  We introduce a comprehensive geolocation framework with three key components.<n>GeoComp, a large-scale dataset; GeoCoT, a novel reasoning method; and GeoEval, an evaluation metric.<n>We demonstrate that GeoCoT significantly boosts geolocation accuracy by up to 25% while enhancing interpretability.
  arXiv  Detail & Related papers  (2025-02-19T14:21:25Z)
• MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models [7.422346909538787]
  We introduce MapEval, a benchmark designed to assess diverse and complex map-based user queries with geo-spatial reasoning.<n>MapEval consists of 700 unique multiple-choice questions about locations across 180 cities and 54 countries.<n>Our detailed analyses provide insights into the strengths and weaknesses of current models, though all models still fall short of human performance by more than 20% on average.<n>This gap highlights MapEval's critical role in advancing general-purpose foundation models with stronger geo-spatial understanding.
  arXiv  Detail & Related papers  (2024-12-31T07:20:32Z)
• Measuring Geographic Diversity of Foundation Models with a Natural Language--based Geo-guessing Experiment on GPT-4 [5.534517268996598]
  We study GPT-4, a state-of-the-art representative in the family of multimodal large language models, to study its geographic diversity. Using DBpedia abstracts as a ground-truth corpus for probing, our natural language-based geo-guessing experiment shows that GPT-4 may currently encode insufficient knowledge about several geographic feature types.
  arXiv  Detail & Related papers  (2024-04-11T09:59:21Z)
• Global-Liar: Factuality of LLMs over Time and Geographic Regions [3.715487408753612]
  This study evaluates the factual accuracy, stability, and biases in widely adopted GPT models, including GPT-3.5 and GPT-4. We introduce 'Global-Liar,' a dataset uniquely balanced in terms of geographic and temporal representation.
  arXiv  Detail & Related papers  (2024-01-31T13:57:24Z)
• Distortions in Judged Spatial Relations in Large Language Models [45.875801135769585]
  GPT-4 exhibited superior performance with 55 percent accuracy, followed by GPT-3.5 at 47 percent, and Llama-2 at 45 percent. The models identified the nearest cardinal direction in most cases, reflecting their associative learning mechanism.
  arXiv  Detail & Related papers  (2024-01-08T20:08:04Z)
• Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition [24.701574433327746]
  We investigate the feasibility of probing a large language model for geography-based object knowledge. We propose geography knowledge regularization to ensure that soft prompts trained on a source set of geographies generalize to an unseen target set. Accuracy gains over prompting baselines on DollarStreet are up to +2.8/1.2/1.6 on target data from Africa/Asia/Americas, and +4.6 overall on the hardest classes.
  arXiv  Detail & Related papers  (2024-01-03T01:11:16Z)
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models [49.20315582673223]
  We present GeoLLM, a novel method that can effectively extract geospatial knowledge from large language models. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe.
  arXiv  Detail & Related papers  (2023-10-10T00:03:23Z)
• Geo-Encoder: A Chunk-Argument Bi-Encoder Framework for Chinese Geographic Re-Ranking [61.60169764507917]
  Chinese geographic re-ranking task aims to find the most relevant addresses among retrieved candidates. We propose an innovative framework, namely Geo-Encoder, to more effectively integrate Chinese geographical semantics into re-ranking pipelines.
  arXiv  Detail & Related papers  (2023-09-04T13:44:50Z)
• GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark [56.08664336835741]
  We propose a GeoGraphic Language Understanding Evaluation benchmark, named GeoGLUE. We collect data from open-released geographic resources and introduce six natural language understanding tasks. We pro vide evaluation experiments and analysis of general baselines, indicating the effectiveness and significance of the GeoGLUE benchmark.
  arXiv  Detail & Related papers  (2023-05-11T03:21:56Z)
• GeoNet: Benchmarking Unsupervised Adaptation across Geographies [71.23141626803287]
  We study the problem of geographic robustness and make three main contributions. First, we introduce a large-scale dataset GeoNet for geographic adaptation. Second, we hypothesize that the major source of domain shifts arise from significant variations in scene context. Third, we conduct an extensive evaluation of several state-of-the-art unsupervised domain adaptation algorithms and architectures.
  arXiv  Detail & Related papers  (2023-03-27T17:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.