Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations

• URL: http://arxiv.org/abs/2605.11633v1
• Date: Tue, 12 May 2026 06:57:41 GMT
• Title: Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
• Abstract summary: Disaster Operational Response Agent benchmark (DORA) is the first agentic benchmark for end-to-end disaster response.<n>Tasks span five dimensions that cover the operational disaster-response pipeline: disaster perception, spatial relational analysis, rescue and evacuation planning, temporal evolution reasoning, and multi-modal report synthesis.<n>DORA establishes a rigorous testbed for operationally reliable disaster-response agents.
• Score: 55.251494694783894
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Operational disaster response goes beyond damage assessment, requiring responders to integrate multi-sensor signals, reason over road networks, populations and key facilities, plan evacuations, and produce actionable reports. However, prior work largely isolates remote-sensing perception or evaluates generic tool use, leaving the end-to-end workflows of emergency operations underexplored. In this paper, we introduce Disaster Operational Response Agent benchmark (DORA), the first agentic benchmark for end-to-end disaster response: 515 expert-authored tasks across 45 real-world disaster events spanning 10 types, paired with expert-verified, replayable gold trajectories totaling 3,500 tool-call steps. Tasks span five dimensions that cover the operational disaster-response pipeline: disaster perception, spatial relational analysis, rescue and evacuation planning, temporal evolution reasoning, and multi-modal report synthesis. Agents compose calls from a 108-tool MCP library over heterogeneous geospatial data: optical, SAR, and multi-spectral imagery across single-, bi-, and multi-temporal sequences (0.015-10m GSD), complemented by elevation and social vector layers. We comprehensively evaluate 13 frontier LLMs on our benchmark, revealing three persistent challenges: 1) disaster-domain grounding exposes unique failure modes (damage-semantic grounding, sensor-modality mismatch, and disaster-pipeline composition); 2) agents are doubly bottlenecked by tool selection and argument grounding, where gold tool-order hints improve accuracy by only 1.08-4.40%, and alternative scaffolds yield at most a 3.24% gain; 3) compositional fragility scales with trajectory length, the agent-to-gold gap widening from 7% to 56% on long pipelines. DORA establishes a rigorous testbed for operationally reliable disaster-response agents.

Related papers

• BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents [20.217145643468598]
  We introduce Blindspot, a benchmark for trajectory-level safety calibration of long-horizon tool-using agents.<n>Blindspot evaluates complete user-agent-environment trajectories through adaptive adversarial interaction, stateful tool execution, and execution-grounded adjudication.<n>We evaluate 13 proprietary and open-weight LLMs using eight metrics covering unsafe completion, appropriate refusal, benign utility, over-refusal, repeated-run robustness, and post-refusal failure.
  arXiv  Detail & Related papers  (2026-09-14T20:12:27Z)
• GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligence [34.091777505490235]
  We introduce GeoDisaster, an operational geospatial disaster reasoning benchmark with 2,921 verified instances across 43 question types.<n>Ground-truth answers are grounded in executable geospatial and deterministic consistency checks.<n>We propose an orchestrated multi-agent framework with 18 disaster-oriented tools, where role-specialized agents coordinate through explicit execution contracts.
  arXiv  Detail & Related papers  (2026-06-15T19:46:10Z)
• Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines [34.37967130277969]
  We propose a two-agent evasion framework operating in a semantic perturbation space.<n>Evasion rates are 19.95 to 40.34% on modern large language model (LLM) based systems.<n>A pattern-informed defense reduces the evasion rate by up to 65.18%.
  arXiv  Detail & Related papers  (2026-04-26T00:58:36Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• RAPTOR-AI for Disaster OODA Loop: Hierarchical Multimodal RAG with Experience-Driven Agentic Decision-Making [0.0]
  This work introduces an agentic Retrieval-Augmented Generation framework designed to support the three canonical phases of disaster response: initial rescue, mid-term recovery, and long-term reconstruction.<n>To achieve robust multimodal grounding, we construct a hierarchical knowledge base that integrates textual disaster manuals, historical lessons, and aerial and ground-level imagery.<n>Our system builds on the open-source multimodal implementation, which processes 46 tsunami-related PDFs (2,378 pages) using BLIP-based image captioning, ColVBERT embeddings, and long-context summarization.
  arXiv  Detail & Related papers  (2026-01-18T10:58:41Z)
• Automating Supply Chain Disruption Monitoring via an Agentic AI Approach [49.77982322940809]
  We introduce a minimally supervised agentic AI framework that autonomously monitors, analyses, and responds to disruptions across extended supply networks.<n>The system achieves high accuracy across core tasks, with F1 scores between 0.962 and 0.991, and performs full end-to-end analyses in a mean of 3.83 minutes at a cost of $0.0836 per disruption.
  arXiv  Detail & Related papers  (2026-01-14T18:28:31Z)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data [76.18834864749606]
  LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm.<n>Existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level.<n>We introduce AuraGen, a controllable engine that synthesizes benign trajectories, injects category-labeled risks with difficulty, and filters outputs via an automated reward model.
  arXiv  Detail & Related papers  (2025-10-10T18:42:32Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks [64.86209459039313]
  ThinkGeo is an agentic benchmark designed to evaluate tool-augmented agents on remote sensing tasks via structured tool use and multi-step planning.<n>We implement a ReAct-style interaction loop and evaluate both open and closed-source LLMs on 486 structured agentic tasks with 1,773 expert-verified reasoning steps.<n>Our analysis reveals notable disparities in tool accuracy and planning consistency across models.
  arXiv  Detail & Related papers  (2025-05-29T17:59:38Z)
• Multi-class Seismic Building Damage Assessment from InSAR Imagery using Quadratic Variational Causal Bayesian Inference [3.190793775376023]
  We present a novel approach to extracting multi-class building damage classifications from InSAR data.<n>Our approach integrates InSAR observations with ground failure models and building functions.<n>Our approach maintains high accuracy (AUC > 0.93) across all damage categories while reducing computational overhead by over 40% without requiring extensive ground truth data.
  arXiv  Detail & Related papers  (2025-02-25T15:40:16Z)
• Deep Self-Supervised Disturbance Mapping with the OPERA Sentinel-1 Radiometric Terrain Corrected SAR Backscatter Product [41.94295877935867]
  Mapping land surface disturbances supports disaster response, resource and ecosystem management, and climate adaptation efforts.<n> Synthetic aperture radar (SAR) is an invaluable tool for disturbance mapping, providing consistent time-series images of the ground regardless of weather or illumination conditions.<n>NASA's Observational Products for End-Users from Remote Sensing Analysis (OPERA) project released the near-global Radiometric Terrain Corrected SAR backscatter from Sentinel-1 (RTC-S1) dataset in October 2023.<n>In this work, we utilize this new dataset to systematically analyze land surface disturbances.
  arXiv  Detail & Related papers  (2025-01-15T20:24:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.