GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models

• URL: http://arxiv.org/abs/2606.12821v1
• Date: Thu, 11 Jun 2026 02:35:20 GMT
• Title: GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
• Abstract summary: We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API.<n>It comprises 93 tasks across 18 categories, covering municipality analysis, multi-turn conversation, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, and task rejection.<n> Tasks are evaluated against an open, self-hostable API serving three environmental indicators across Spain and Portugal via sixteen tools.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Environmental scientists spend disproportionate effort on data wrangling rather than analysis, and AI agents that automate geospatial workflows remain unvalidated: no benchmark evaluates agents operating through structured tool calling against real APIs. We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API. It comprises 93 tasks across 18 categories, covering municipality analysis, multi-turn conversation, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, and task rejection. Tasks are evaluated against an open, self-hostable API serving three environmental indicators across Spain and Portugal via sixteen tools. We evaluate seven LLMs (Claude Sonnet 4, DeepSeek V3.2, GLM-5, Gemini 2.5 Pro, Qwen3-235B, GPT-OSS-120B, Llama 4 Scout) under three temperature-1.0 seeds, reporting capability and per-case cost as orthogonal axes. We find: (1) Claude Sonnet 4 leads at 60.8% +/- 0.8%, followed by DeepSeek V3.2 at 56.3% +/- 3.1%, with no other model above 51%; (2) the cost-accuracy Pareto frontier is occupied mostly by open-weight models, with DeepSeek V3.2 offering 93% of Claude's capability at 11x lower cost ($0.011/case); (3) comparison tasks remain universally unsolved (0% on close-value comparisons), exposing systematic reasoning limits; and (4) structured tool calling against a real API is more discriminative than general-purpose GIS benchmarks, with accuracies 25-35 points lower. We further show extensibility by integrating BigEarthNet V2 land cover for Portugal alongside Spanish CO2 and erosion indicators. The benchmark, harness, and self-hostable API are publicly available.

Related papers

• Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent [98.33217711280383]
  We introduce Agents-A1, a 35B Mixture-of-Experts Agentic Model that reaches trillion- parameter-level performance by scaling the agent horizon.<n>We build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes.<n>Agents-A1 achieves strong and broad performance for long-horizon agent benchmarks.
  arXiv  Detail & Related papers  (2026-06-29T17:50:54Z)
• InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy [0.9924185669370708]
  InvestPhilBench is a dynamic benchmark spanning eight cognitive tiers.<n>The v0.6 release comprises 118 primary-source-verified investment principle cards.<n>For reproducible scoring at scale we introduce the Benchmark Automated Scoring Pipeline.
  arXiv  Detail & Related papers  (2026-06-24T15:53:20Z)
• HydroAgent: Closing the Gap Between Frontier LLMs and Human Experts in Hydrologic Model Calibration via Simulator-Grounded RL [6.680830184790049]
  Calibrating distributed hydrologic models is a critical bottleneck across operational water resources management.<n>We ask: can frontier large language model (LLM) agents replace the human hydrologic modeler, and if not, what would it take?<n>We benchmark nine frontier LLM agents on the operational CREST distributed hydrologic model used by the U.S. National Weather Service for flash-flood forecasting.
  arXiv  Detail & Related papers  (2026-05-18T03:17:38Z)
• PersonalAI 2.0: Enhancing knowledge graph traversal/retrieval with planning mechanism for Personalized LLM Agents [53.369140518949656]
  We introduce PersonalAI 2.0 (PAI-2), a novel framework to enhance large language model (LLM) based systems through integration of external knowledge graphs (KG)<n>The proposed approach addresses key limitations of existing Retrieval-Augmented Generation (GraphRAG) methods by incorporating a dynamic, multistage query processing pipeline.<n>We show that use of graph algorithms (e.g. BeamSearch, WaterCircles) gain superior results compared to standard flatten retriever on average 6%, while enabled search plan enhancement mechanism gain 18% boost compared to disabled one by LLM-as-a-Judge across six datasets.
  arXiv  Detail & Related papers  (2026-05-13T13:06:30Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Governance Architecture for Autonomous Agent Systems: Threats, Framework, and Engineering Practice [0.0]
  We propose the Layered Governance Architecture (LGA), a four-layer framework comprising execution sandboxing (L1), intent verification (L2), zero-trust inter-agent authorization (L3), and immutable audit logging (L4)<n>To evaluate LGA, we construct a benchmark (Chinese original, English via machine translation) of 1,081 tool-call samples -- covering prompt injection, RAG poisoning, and malicious skill plugins -- and apply it to OpenClaw, a representative open-source agent framework.
  arXiv  Detail & Related papers  (2026-03-07T13:05:14Z)
• FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents [3.5621895565039203]
  FinRetrieval is a benchmark of 500 financial retrieval questions with ground truth answers.<n>Claude Opus achieves 90.8% accuracy with structured data APIs but only 19.8% with web search alone.
  arXiv  Detail & Related papers  (2026-01-02T17:51:37Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models [194.64264251080454]
  We present GLM-4.5, an open-source Mixture-of-Experts (MoE) large language model with 355B total parameters and 32B activated parameters.<n>Through multi-stage training on 23T tokens and comprehensive post-training with expert model iteration and reinforcement learning, GLM-4.5 achieves strong performance across agentic, reasoning, and coding tasks.<n>We release both GLM-4.5 (355B parameters) and a compact version, GLM-4.5-Air (106B parameters), to advance research in reasoning and agentic AI systems.
  arXiv  Detail & Related papers  (2025-08-08T17:21:06Z)
• Benchmarking Large Language Models for Geolocating Colonial Virginia Land Grants [0.0]
  Virginia's seventeenth- and eighteenth-century land patents survive primarily as narrative metes-and-bounds descriptions.<n>This study systematically evaluates current-generation large language models (LLMs) in converting these prose abstracts into geographically accurate latitude/longitude coordinates.
  arXiv  Detail & Related papers  (2025-07-27T21:49:58Z)
• GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks [0.11458853556386796]
  This paper establishes a benchmark for evaluating tool-calling capabilities of large language models (LLMs)<n>We assess eight commercial LLMs (Claude Sonnet 3.5 and 4, Claude Haiku 3.5, Gemini 2.0 Flash, Gemini 2.5 Pro Preview, GPT-4o, GPT-4.1 and o4-mini) using a simple tool-calling agent equipped with 23 geospatial functions.<n>Results show o4-mini and Claude 3.5 Sonnet achieve the best overall performance, OpenAI's GPT-4.1, GPT-4o and Google's Gemini 2.5 Pro Preview do not fall far behind, but the last two are more efficient in
  arXiv  Detail & Related papers  (2025-03-23T16:20:14Z)
• Distortions in Judged Spatial Relations in Large Language Models [45.875801135769585]
  GPT-4 exhibited superior performance with 55 percent accuracy, followed by GPT-3.5 at 47 percent, and Llama-2 at 45 percent. The models identified the nearest cardinal direction in most cases, reflecting their associative learning mechanism.
  arXiv  Detail & Related papers  (2024-01-08T20:08:04Z)
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models [49.20315582673223]
  We present GeoLLM, a novel method that can effectively extract geospatial knowledge from large language models. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe.
  arXiv  Detail & Related papers  (2023-10-10T00:03:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.