GeoR-Bench: Evaluating Geoscience Visual Reasoning

• URL: http://arxiv.org/abs/2605.11541v1
• Date: Tue, 12 May 2026 05:13:37 GMT
• Title: GeoR-Bench: Evaluating Geoscience Visual Reasoning
• Abstract summary: It remains unclear how far current AI systems are from achieving genuine geoscience intelligence.<n>We present textbfGeoR-Bench, a underlineBenchmark for evaluating underlineGeoscience visual underlineReasoning through reasoning informed visual editing tasks.
• Score: 102.4693880357753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Geoscience intelligence is expected to understand, reason about, and predict earth system changes to support human decision-making in critical domains such as disaster response, climate adaptation and environmental protection. Although current research has shown promising progress on specific geoscience tasks, such as remote sensing interpretation, geographic question-answering, existing benchmarks remain largely task-specific which failing to capture the open-ended real world geoscience problems. As a result, it remains unclear how far current AI systems are from achieving genuine geoscience intelligence. To address this gap, we present \textbf{GeoR-Bench}, a \underline{Bench}mark for evaluating \underline{Geo}science visual \underline{R}easoning through reasoning informed visual editing tasks. GeoR-Bench contains 440 curated samples spanning 6 geoscience categories and 24 task types, covering earth observation imagery and structured scientific representations such as maps and diagrams. We evaluate outputs along three dimensions, including reasoning, consistency, and quality. Benchmark results of 21 closed- and open-source multimodal models reveal that geoscience reasoning remains a critical bottleneck. The highest-performing model achieves 42.7\% overall strict accuracy, while the best open-source models only get 10.3\%. Notably, the visual consistency and image quality of the outputs frequently surpass their scientific accuracy. Ultimately, these findings indicate that current models generate superficially plausible results but fail to capture underlying earth science processes.

Related papers

• GeoAgent: Learning to Geolocate Everywhere with Reinforced Geographic Characteristics [91.17301794848025]
  This paper presents GeoAgent, a model capable of reasoning closely with humans and deriving fine-grained address conclusions.<n>Previous RL-based methods have achieved breakthroughs in performance and interpretability but still remain concerns because of their reliance on AI-generated chain-of-thought (CoT) data and training strategies.
  arXiv  Detail & Related papers  (2026-02-13T04:48:05Z)
• Geo3DVQA: Evaluating Vision-Language Models for 3D Geospatial Reasoning from Aerial Imagery [18.7420518276348]
  Geo3DVQA is a benchmark for evaluating vision-language models (VLMs) in height-aware, 3D geospatial reasoning.<n>Unlike conventional sensor-based frameworks, Geo3DVQA emphasizes realistic scenarios that integrate elevation, sky view factors, and land cover patterns.
  arXiv  Detail & Related papers  (2025-12-08T08:16:14Z)
• GeoBS: Information-Theoretic Quantification of Geographic Bias in AI Models [34.611626290720295]
  We establish an information-theoretic framework for geo-bias evaluation, called GeoBS (Geo-Bias Scores)<n>We propose three novel geo-bias scores that explicitly take intricate spatial factors into consideration.
  arXiv  Detail & Related papers  (2025-09-27T20:07:21Z)
• GeoEvolve: Automating Geospatial Model Discovery via Multi-Agent Large Language Models [49.257706111340134]
  We introduce GeoEvolve, a multi-agent LLM framework that couples evolutionary search with geospatial domain knowledge.<n>We evaluate it on two fundamental and classical tasks: spatial (kriging) and spatial uncertainty.<n>It reduces spatial error (RMSE) by 13-21% and enhances uncertainty estimation performance by 17%.
  arXiv  Detail & Related papers  (2025-09-25T21:03:57Z)
• RAG for Geoscience: What We Expect, Gaps and Opportunities [15.069356714106808]
  Retrieval-Augmented Generation (RAG) enhances language models by combining retrieval with generation.<n>We envision Geo-RAG, a next-generation paradigm that reimagines RAG as a modular retrieve $rightarrow$ reason $rightarrow$ generate $rightarrow$ verify loop.<n>Geo-RAG supports four core capabilities: (i) retrieval of multi-modal Earth data; (ii) reasoning under physical and domain constraints; (iii) generation of science-grade artifacts; and (iv) verification of generated hypotheses against numerical models, ground measurements, and expert assessments.
  arXiv  Detail & Related papers  (2025-08-15T06:33:27Z)
• GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data? [5.017671236021897]
  GeoGrid-Bench is a benchmark designed to evaluate the ability of foundation models to understand geo-spatial data in the grid structure.<n>This benchmark features large-scale, real-world data covering 16 climate variables across 150 locations and extended time frames.
  arXiv  Detail & Related papers  (2025-05-15T21:31:44Z)
• Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework [59.42946541163632]
  We introduce a comprehensive geolocation framework with three key components.<n>GeoComp, a large-scale dataset; GeoCoT, a novel reasoning method; and GeoEval, an evaluation metric.<n>We demonstrate that GeoCoT significantly boosts geolocation accuracy by up to 25% while enhancing interpretability.
  arXiv  Detail & Related papers  (2025-02-19T14:21:25Z)
• PEACE: Empowering Geologic Map Holistic Understanding with MLLMs [64.58959634712215]
  Geologic map, as a fundamental diagram in geology science, provides critical insights into the structure and composition of Earth's subsurface and surface.<n>Despite their significance, current Multimodal Large Language Models (MLLMs) often fall short in geologic map understanding.<n>To quantify this gap, we construct GeoMap-Bench, the first-ever benchmark for evaluating MLLMs in geologic map understanding.
  arXiv  Detail & Related papers  (2025-01-10T18:59:42Z)
• GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark [56.08664336835741]
  We propose a GeoGraphic Language Understanding Evaluation benchmark, named GeoGLUE. We collect data from open-released geographic resources and introduce six natural language understanding tasks. We pro vide evaluation experiments and analysis of general baselines, indicating the effectiveness and significance of the GeoGLUE benchmark.
  arXiv  Detail & Related papers  (2023-05-11T03:21:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.