Related papers: TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route

TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route

URL: http://arxiv.org/abs/2509.18173v1
Date: Wed, 17 Sep 2025 15:00:03 GMT
Title: TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route
Authors: Hongyi Luo, Qing Cheng, Daniel Matos, Hari Krishna Gadi, Yanfeng Zhang, Lu Liu, Yongliang Wang, Niclas Zeller, Daniel Cremers, Liqiu Meng,
Abstract summary: We create a large-scale evaluation dataset comprised of 36000 routes from 12 metropolises worldwide.<n>We introduce PathBuilder, a novel tool for converting natural language instructions into navigation routes.<n>We rigorously assess 11 state-of-the-art (SOTA) LLMs on the task of route reversal.
Score: 45.16008377814563
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Humans can interpret geospatial information through natural language, while the geospatial cognition capabilities of Large Language Models (LLMs) remain underexplored. Prior research in this domain has been constrained by non-quantifiable metrics, limited evaluation datasets and unclear research hierarchies. Therefore, we propose a large-scale benchmark and conduct a comprehensive evaluation of the geospatial route cognition of LLMs. We create a large-scale evaluation dataset comprised of 36000 routes from 12 metropolises worldwide. Then, we introduce PathBuilder, a novel tool for converting natural language instructions into navigation routes, and vice versa, bridging the gap between geospatial information and natural language. Finally, we propose a new evaluation framework and metrics to rigorously assess 11 state-of-the-art (SOTA) LLMs on the task of route reversal. The benchmark reveals that LLMs exhibit limitation to reverse routes: most reverse routes neither return to the starting point nor are similar to the optimal route. Additionally, LLMs face challenges such as low robustness in route generation and high confidence for their incorrect answers. Code\ \&\ Data available here: \href{https://github.com/bghjmn32/EMNLP2025_Turnback}{TurnBack.}

Related papers

Enhancing Geometric Perception in VLMs via Translator-Guided Reinforcement Learning [52.075928878249066]
Vision-guided models (VLMs) often struggle with geometric reasoning due to their limited perception of fundamental diagram elements.<n>We introduce GeoPerceive, a benchmark comprising diagram instances paired with domain-specific language representations.<n>We propose GeoDPO, a translator reinforcement learning framework.
arXiv Detail & Related papers (2026-02-26T07:28:04Z)
GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes [84.52881742231152]
Multimodal large language models (MLLMs) have undergone rapid development in advancing geospatial scene understanding.<n>Recent studies have sought to enhance the reasoning capabilities of remote sensing MLLMs, typically through cold-start training with elaborately curated chain-of-thought (CoT) data.<n>We propose GeoZero, a framework that enables MLLMs to perform geospatial reasoning without any predefined CoT supervision.
arXiv Detail & Related papers (2025-11-27T17:28:09Z)
CompassLLM: A Multi-Agent Approach toward Geo-Spatial Reasoning for Popular Path Query [10.085519288797345]
We introduce CompassLLM, a novel multi-agent framework to solve the popular path query.<n> CompassLLM employs its agents in a two-stage pipeline: the SEARCH stage that identifies popular paths, and the GENERATE stage that synthesizes novel paths in the absence of an existing one in the historical trajectory data.
arXiv Detail & Related papers (2025-10-08T20:28:52Z)
Understanding the Geospatial Reasoning Capabilities of LLMs: A Trajectory Recovery Perspective [31.228269455751363]
This paper explores whether Large Language Models (LLMs) can read road network maps and perform navigation.<n>We frame trajectory recovery as a proxy task, which requires models to reconstruct masked GPS traces.<n>Using road network as context, our prompting framework enables LLMs to generate valid paths without accessing any external navigation tools.
arXiv Detail & Related papers (2025-10-02T03:37:41Z)
RALLM-POI: Retrieval-Augmented LLM for Zero-shot Next POI Recommendation with Geographical Reranking [7.085868567930685]
Next point-of-interest (POI) recommendation predicts a user's next destination from historical movements.<n>Traditional models require intensive training, while LLMs offer flexible and generalizable zero-shot solutions.<n>We propose RALLM-POI, a framework that couples LLMs with retrieval-augmented generation and self-rectification.
arXiv Detail & Related papers (2025-09-21T12:52:28Z)
RoadMind: Towards a Geospatial AI Expert for Disaster Response [6.038207189709353]
Large Language Models (LLMs) have shown impressive performance across a range of natural language tasks, but remain limited in their ability to reason about geospatial data.<n>We present RoadMind, a self-supervised framework that enhances the geospatial reasoning capabilities of LLMs using structured data from OpenStreetMap (OSM)<n>Our results show that models trained via RoadMind significantly outperform strong baselines, including state-of-the-art LLMs equipped with advanced prompt engineering.
arXiv Detail & Related papers (2025-09-18T09:46:55Z)
LLMAP: LLM-Assisted Multi-Objective Route Planning with User Preferences [31.10423199218523]
The rise of large language models (LLMs) has made natural language-driven route planning an emerging research area that encompasses rich user objectives.<n>In this paper, we introduce a novel LLM-as task to comprehend natural language, identify tasks, and extract user preferences.<n>We conduct extensive experiments using 1,000 routing prompts sampled with varying complexity across 14 countries and 27 cities worldwide.
arXiv Detail & Related papers (2025-09-14T02:30:19Z)
Unlocking Location Intelligence: A Survey from Deep Learning to The LLM Era [12.411524513969603]
Location Intelligence (LI) is the science of transforming location-centric geospatial data into actionable knowledge.<n>The rapid evolution of Geospatial Representation Learning is fundamentally reshaping LI development through two successive technological revolutions.<n>This survey presents a comprehensive review of geospatial representation learning across both technological eras.
arXiv Detail & Related papers (2025-05-13T12:16:26Z)
Reliable, Adaptable, and Attributable Language Models with Retrieval [144.26890121729514]
Parametric language models (LMs) are trained on vast amounts of web data. They face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. We advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs.
arXiv Detail & Related papers (2024-03-05T18:22:33Z)
GeoLLM: Extracting Geospatial Knowledge from Large Language Models [49.20315582673223]
We present GeoLLM, a novel method that can effectively extract geospatial knowledge from large language models. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe.
arXiv Detail & Related papers (2023-10-10T00:03:23Z)
MGeo: Multi-Modal Geographic Pre-Training Method [49.78466122982627]
We propose a novel query-POI matching method Multi-modal Geographic language model (MGeo) MGeo represents GC as a new modality and is able to fully extract multi-modal correlations for accurate query-POI matching. Our proposed multi-modal pre-training method can significantly improve the query-POI matching capability of generic PTMs.
arXiv Detail & Related papers (2023-01-11T03:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.