Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System

• URL: http://arxiv.org/abs/2608.11738v1
• Date: Wed, 12 Aug 2026 07:25:32 GMT
• Title: Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System
• Abstract summary: UAV-MAS is a training-free multi-agent system for MLLM-based UAV aerial image understanding and reasoning.<n>It achieves 77.0% overall accuracy on UAVQA-Bench, surpassing Gemini 3 Pro by 4.0%, while the 8B variant improves 8.7% over its base model.
• Score: 21.596261657771
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Model (MLLM)-based UAV aerial image understanding and reasoning is essential for aerial intelligence yet poses distinct challenges arising from extreme scale variation, arbitrary camera orientations, and high object density. Despite growing interest, existing evaluations remain fragmented across individual datasets and narrow tasks, leaving a critical gap in unified assessment of UAV understanding and reasoning capabilities. To fill this gap, we construct UAVQA-Bench, a benchmark of 1,500 human-annotated QA pairs drawn from 13 public UAV datasets, covering 6 capability dimensions and 16 tasks in both multiple-choice and visual grounding formats. Systematic evaluation of a broad range of open-source and closed-source MLLMs as well as agent-based systems on UAVQA-Bench identifies three key failure modes: domain-toolset mismatch, unchecked error propagation, and static reasoning. Motivated by these findings, we propose UAV-MAS, a training-free multi-agent system for MLLM-based UAV aerial image understanding and reasoning, comprising a Domain-Specific Perception Engine (DSPE) that routes queries to task-appropriate visual tools, a Context-Aware Iterative Refinement module (CAIR) that validates intermediate reasoning to curb error accumulation, and a Difficulty-Aware Adaptive Search mechanism (DAAS) that adjusts search depth to question difficulty. UAV-MAS with a 32B open-source MLLM achieves 77.0% overall accuracy on UAVQA-Bench, surpassing Gemini 3 Pro by 4.0\%, while the 8B variant improves 8.7\% over its base model.

Related papers

• Multimodal Model Diffing for Feature Discovery and Control [25.349396112139214]
  We introduce MMDiff, a framework that turns sparse autoencoders into feature-level interfaces for discovering and controlling multimodal behavior.<n>We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR.
  arXiv  Detail & Related papers  (2026-08-10T17:59:30Z)
• MultiUAV-Plat: An LLM-Oriented Platform, Benchmark and Framework for Multi-UAV Collaborative Task Planning [4.712457848623237]
  Large language models (LLMs) provide a promising interface for high-level robotic task planning.<n>Existing UAV simulators mainly emphasize dynamics, perception, or low-level control.<n>Existing LLM-agents rarely capture aerial-robotics constraints such as partial observability.<n>We present MultiUAV-Plat, a lightweight, easy-to-use simulation platform for multi-UAV task planning.
  arXiv  Detail & Related papers  (2026-06-30T03:02:12Z)
• SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion [76.8430169655641]
  spatial intelligence is essential for low-altitude unmanned aerial vehicle (UAV) perception, collaboration, and navigation.<n>We introduce SpatialUAV, a real low-altitude UAV benchmark covering semantic discrimination, spatial relation, aerial-aerial collaboration, aerial-ground collaboration, and motion understanding.<n>We show that current models remain far from human-level performance, with pronounced bottlenecks in cross-view association, structured grounding, geometric reasoning, and temporal viewpoint understanding.
  arXiv  Detail & Related papers  (2026-06-26T09:16:55Z)
• PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation [71.53011159355401]
  Reasoning segmentation has expanded from ground-level scenes to remote-sensing imagery, yet UAV data poses distinct challenges.<n>We define the UAV Reasoning task and organize its semantic requirements into three dimensions: Spatial, Attribute, and Scene-level reasoning.<n>We construct DRSeg, a large-scale benchmark for UAV reasoning segmentation, containing 10k high-resolution aerial images paired with Chain-of-Thought QA supervision.
  arXiv  Detail & Related papers  (2026-04-17T03:48:56Z)
• M3-AD: Reflection-aware Multi-modal, Multi-category, and Multi-dimensional Benchmark and Framework for Industrial Anomaly Detection [87.27067827385599]
  M3-AD is a unified reflection-aware framework for industrial anomaly detection.<n> RA-Monitor guides models to perform controlled self-correction when initial judgments are unreliable.
  arXiv  Detail & Related papers  (2026-02-10T07:20:19Z)
• MM-UAVBench: How Well Do Multimodal Large Language Models See, Think, and Plan in Low-Altitude UAV Scenarios? [35.75859316774549]
  We present MM-UAVBench, a comprehensive benchmark that systematically evaluates MLLMs across three core capability dimensions-perception, cognition, and planning-in low-altitude UAV scenarios.<n> MM-UAVBench comprises 19 sub-tasks with over 5.7K manually annotated questions, all derived from real-world UAV data collected from public datasets.<n>Our experiments reveal that current models struggle to adapt to the complex visual and cognitive demands of low-altitude scenarios.
  arXiv  Detail & Related papers  (2025-12-29T05:49:54Z)
• AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios [64.51320327698231]
  We introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios.<n>We develop an innovative semi-automated collaborative agent-based labeling assistant framework.<n>We also propose HawkEyeTrack, a novel method that collaboratively enhances vision-language representation learning.
  arXiv  Detail & Related papers  (2025-11-26T04:44:27Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• UAVBench: An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios [3.099103925863002]
  UAVBench is an open benchmarking benchmark for UAV flight scenarios generated through large language models (LLMs)<n>We present UAVBench_MCQ, a reasoning-oriented extension containing 50,000 multiple-choice questions spanning ten cognitive and ethical reasoning styles.<n>We evaluate 32 state-of-the-art LLMs, including GPT-5, ChatGPT-4o, Gemini 2.5 Flash, DeepSeek V3, Q3wenwenB, and ERNIE 4.5 300B, and find strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making.
  arXiv  Detail & Related papers  (2025-11-14T12:51:48Z)
• Agentic UAVs: LLM-Driven Autonomy with Integrated Tool-Calling and Cognitive Reasoning [3.4643961367503575]
  Existing UAV frameworks lack context-aware reasoning, autonomous decision-making, and ecosystem-level integration.<n>This paper introduces the Agentic UAVs framework, a five-layer architecture (Perception, Reasoning, Action, Integration, Learning)<n>A ROS2 and Gazebo-based prototype integrates YOLOv11 object detection with GPT-4 reasoning and local Gemma-3 deployment.
  arXiv  Detail & Related papers  (2025-09-14T08:46:40Z)
• Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation [88.78166077081912]
  We introduce a multimodal unlearning benchmark, UnLOK-VQA, and an attack-and-defense framework to evaluate methods for deleting specific multimodal knowledge from MLLMs.<n>Our results show multimodal attacks outperform text- or image-only ones, and that the most effective defense removes answer information from internal model states.
  arXiv  Detail & Related papers  (2025-05-01T01:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.