Benchmarking Large Language Models for Geolocating Colonial Virginia Land Grants

• URL: http://arxiv.org/abs/2508.08266v1
• Date: Sun, 27 Jul 2025 21:49:58 GMT
• Title: Benchmarking Large Language Models for Geolocating Colonial Virginia Land Grants
• Abstract summary: Virginia's seventeenth- and eighteenth-century land patents survive primarily as narrative metes-and-bounds descriptions.<n>This study systematically evaluates current-generation large language models (LLMs) in converting these prose abstracts into geographically accurate latitude/longitude coordinates.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Virginia's seventeenth- and eighteenth-century land patents survive primarily as narrative metes-and-bounds descriptions, limiting spatial analysis. This study systematically evaluates current-generation large language models (LLMs) in converting these prose abstracts into geographically accurate latitude/longitude coordinates within a focused evaluation context. A digitized corpus of 5,471 Virginia patent abstracts (1695-1732) is released, with 43 rigorously verified test cases serving as an initial, geographically focused benchmark. Six OpenAI models across three architectures (o-series, GPT-4-class, and GPT-3.5) were tested under two paradigms: direct-to-coordinate and tool-augmented chain-of-thought invoking external geocoding APIs. Results were compared with a GIS-analyst baseline, the Stanford NER geoparser, Mordecai-3, and a county-centroid heuristic. The top single-call model, o3-2025-04-16, achieved a mean error of 23 km (median 14 km), outperforming the median LLM (37.4 km) by 37.5%, the weakest LLM (50.3 km) by 53.5%, and external baselines by 67% (GIS analyst) and 70% (Stanford NER). A five-call ensemble further reduced errors to 19 km (median 12 km) at minimal additional cost (approx. USD 0.20 per grant), outperforming the median LLM by 48.6%. A patentee-name-redaction ablation increased error by about 9%, indicating reliance on textual landmark and adjacency descriptions rather than memorization. The cost-efficient gpt-4o-2024-08-06 model maintained a 28 km mean error at USD 1.09 per 1,000 grants, establishing a strong cost-accuracy benchmark; external geocoding tools offered no measurable benefit in this evaluation. These findings demonstrate the potential of LLMs for scalable, accurate, and cost-effective historical georeferencing.

Related papers

• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• GeoAI-based post-segmentation quality validation of building footprints via spatial feature engineering [1.3277276618043652]
  Deep learning-based building footprint extraction often produces top inconsistent vectors for direct GIS database ingestion.<n>We present a multidomain GeoAI quality control framework that automates error detection to purify vector footprint databases.<n>This framework successfully identified 87.34% of erroneous footprints while maintaining 98.31% of acceptable structures.
  arXiv  Detail & Related papers  (2026-08-10T02:55:07Z)
• GeoID-PINN: Identifiability-Aware Regional Epidemic Inference with Geographic Coupling [1.0875426095207128]
  We introduce a physics-informed neural network (PINN) for susceptible-infectious-recovered-deceased dynamics.<n>We regularize a spatial prior constructed from distance, adjacency, commuting, or lead-lag information.<n>Relative to an autoregressive negative-binomial baseline, Forecast-Trained Geo-PINN reduces mean squared error (MSE) from 32,957 to 11,468 and mean absolute error (MAE) from 70.60 to 57.73.
  arXiv  Detail & Related papers  (2026-07-28T05:59:23Z)
• GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models [0.0]
  We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API.<n>It comprises 93 tasks across 18 categories, covering municipality analysis, multi-turn conversation, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, and task rejection.<n> Tasks are evaluated against an open, self-hostable API serving three environmental indicators across Spain and Portugal via sixteen tools.
  arXiv  Detail & Related papers  (2026-06-11T02:35:20Z)
• Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs [48.83701310501069]
  Large language models (LLMs) show strong potential for neural architecture generation.<n>Existing approaches produce complete model implementations from scratch.<n>We propose Delta-Code Generation, where fine-tuned LLMs generate compact unified diffs.
  arXiv  Detail & Related papers  (2026-05-06T13:32:05Z)
• Physically Interpretable AlphaEarth Foundation Model Embeddings Enable LLM-Based Land Surface Intelligence [0.0]
  We present a comprehensive interpretability analysis of Google AlphaEarth's 64-dimensional embeddings against 26 environmental variables.<n>We then developed a Land Surface Intelligence system that implements retrieval-augmented generation over a FAISS-indexed embedding database of 12.1 million vectors.
  arXiv  Detail & Related papers  (2026-02-10T22:58:50Z)
• Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales [61.03549470159347]
  Vision-language models (VLMs) have advanced rapidly, yet their capacity for image-grounded geolocation in open-world conditions has not been comprehensively evaluated.<n>We present EarthWhere, a comprehensive benchmark for VLM image geolocation that evaluates visual recognition, step-by-step reasoning, and evidence use.
  arXiv  Detail & Related papers  (2025-10-13T01:12:21Z)
• UrbanScore: A Real-Time Personalised Liveability Analytics Platform [0.0]
  UrbanScore is a real-time web platform that computes a personalised liveability score for any urban address.<n>System fuses five data streams: address geocoding via Nominatim, (ii) facility extraction from OpenStreetMap through Overpass QL, (iii) segment-level traffic metrics from TomTom Flow v10, (iv) hourly air-quality readings from OpenWeatherMap, and (v) user-declared preference profiles.
  arXiv  Detail & Related papers  (2025-07-16T08:53:11Z)
• The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features [0.0]
  This study evaluates the performance of GPT-4o and Gemini 2.0 Flash in three key geospatial tasks: geocoding, elevation estimation, and reverse geocoding.<n>Neither model achieved an accurate reconstruction of Austria's federal states, highlighting persistent misclassifications.
  arXiv  Detail & Related papers  (2025-05-30T20:14:17Z)
• Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval [49.1574468325115]
  We introduce Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones.<n>Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10.<n>More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M) remain competitive while being over 13x smaller.
  arXiv  Detail & Related papers  (2025-05-25T23:06:20Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models [7.422346909538787]
  MapEval is a benchmark designed to assess foundation models across three distinct tasks.<n>It covers spatial relationships, navigation, travel planning, and real-world map interactions.<n>It requires models to handle long-context reasoning, API interactions, and visual map analysis.
  arXiv  Detail & Related papers  (2024-12-31T07:20:32Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)
• Distortions in Judged Spatial Relations in Large Language Models [45.875801135769585]
  GPT-4 exhibited superior performance with 55 percent accuracy, followed by GPT-3.5 at 47 percent, and Llama-2 at 45 percent. The models identified the nearest cardinal direction in most cases, reflecting their associative learning mechanism.
  arXiv  Detail & Related papers  (2024-01-08T20:08:04Z)
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models [49.20315582673223]
  We present GeoLLM, a novel method that can effectively extract geospatial knowledge from large language models. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe.
  arXiv  Detail & Related papers  (2023-10-10T00:03:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.