EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards

• URL: http://arxiv.org/abs/2608.23525v1
• Date: Mon, 24 Aug 2026 17:29:16 GMT
• Title: EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards
• Abstract summary: We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations.<n>Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer.<n>Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%.
• Score: 73.12811119273206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.

Related papers

• TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents [11.367029696505357]
  We present TruthInsightBench, a benchmark configured for discovery.<n>Its 40 blind tasks, drawn from 40 peer-reviewed studies across 10 scientific domains, expose only a neutral scientific objective and frozen data.<n>A fixed LLM-based judge scores the evidentiary maturity of an agent's own claims along six dimensions.
  arXiv  Detail & Related papers  (2026-09-04T12:37:20Z)
• Scaling Scientific Discovery Environments for Turn-Level Agentic RL [14.746461766950162]
  This paper introduces SciDisco, a framework for training Scientific Discovery agents in process-verifiable environments.<n>SciThque compiles hypotheses, datasets, hidden evidence graphs, and verifiers into task environments where analytical progress can be checked.<n>Experiments show that SciDisco-14B reaches state-of-the-art on hypothesis-driven scientific data analysis benchmarks.
  arXiv  Detail & Related papers  (2026-07-31T03:34:39Z)
• TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data? [32.95801052071361]
  We introduce TerraBench, a benchmark for grounded Earth-science reasoning.<n>It is the first in this space to pair process-level tool-use metrics with tolerance-aware numeric scoring.<n>The benchmark comprises 403 extensive agentic tasks across three tracks.
  arXiv  Detail & Related papers  (2026-06-11T10:26:03Z)
• Science Earth: Towards A Planet-Scale Operating System for AI-Native Scientific Discovery [51.70607714288616]
  Science Earth is a planet-scale scientific runtime in which any capability can connect to any other.<n>Its underlying EACN protocol lets capabilities discover one another, negotiate task ownership, and adjudicate across incompatible evidentiary standards.<n>Two runs validate this under structurally distinct conditions.
  arXiv  Detail & Related papers  (2026-05-31T16:05:41Z)
• CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists [28.253879252786632]
  We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents.<n>Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism.
  arXiv  Detail & Related papers  (2026-05-25T16:57:06Z)
• KISS - Knowledge Infrastructure for Scientific Simulation: A Scaffolding for Agentic Earth Science [9.554707285564593]
  Knowledge infrastructure (KI) externalizes expertise into validated modelling operators.<n>Agents equipped with KI produced physically plausible, verifiable end-to-end simulations in up to 84% of trials.<n>Across all 119 KIs, modelling decisions and failure remedies converged despite different underlying physics.
  arXiv  Detail & Related papers  (2026-05-18T04:57:25Z)
• AI scientists produce results without reasoning scientifically [3.100302590436282]
  Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously.<n>Here, we evaluate LLM-based scientific agents across eight domains, spanning execution to hypothesis-driven inquiry.
  arXiv  Detail & Related papers  (2026-04-20T20:23:42Z)
• Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents [49.3216026940601]
  Earth observation is essential for understanding the states of the Earth system.<n>Recent MLLMs have advanced EO research, but they still lack the capability to tackle complex tasks that require multi-step reasoning.<n>We introduce Earth-Agent, the first agentic framework that unifies RGB and spectral EO data within an MCP-based tool ecosystem.
  arXiv  Detail & Related papers  (2025-09-27T06:04:28Z)
• A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [251.23085679210206]
  Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
  arXiv  Detail & Related papers  (2025-08-28T18:30:52Z)
• A Self-Evolving AI Agent System for Climate Science [59.08800209508371]
  We introduce EarthLink, the first self-evolving AI agent system designed as an interactive "copilot" for Earth scientists.<n>Through natural language interaction, EarthLink automates the entire research workflow by integrating planning, code execution, data analysis, and physical reasoning.<n>It exhibits human-like cross-disciplinary analytical ability and proficiency comparable to a junior researcher in expert evaluations on core large-scale climate tasks.
  arXiv  Detail & Related papers  (2025-07-23T08:29:25Z)
• Causal Representation Learning in Temporal Data via Single-Parent Decoding [66.34294989334728]
  Scientific research often seeks to understand the causal structure underlying high-level variables in a system. Scientists typically collect low-level measurements, such as geographically distributed temperature readings. We propose a differentiable method, Causal Discovery with Single-parent Decoding, that simultaneously learns the underlying latents and a causal graph over them.
  arXiv  Detail & Related papers  (2024-10-09T15:57:50Z)
• Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
  This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations.<n>We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.<n>Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
  arXiv  Detail & Related papers  (2024-05-27T13:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.