A Systems Engineering Framework for Vision-Language-Enabled UAV Triage and Disaster Response

• URL: http://arxiv.org/abs/2607.27597v1
• Date: Thu, 30 Jul 2026 02:35:39 GMT
• Title: A Systems Engineering Framework for Vision-Language-Enabled UAV Triage and Disaster Response
• Abstract summary: This study explores the viability of embedding Vision Language Models (VLMs) as coordination agents within the human-UAV loop.<n>The proposed architecture integrates natural language interaction, mission-level task coordination, software-in-the-loop implementation, and communication aligned with the Incident Command System (ICS)<n>Three key elements, the VLM Coordinator Agent, UAV Mission Control, and Task Allocator, were implemented within an integrated simulation and control environment.
• Score: 1.8479315677380457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Vision Language Models (VLMs) have created new opportunities for disaster response, where responders must interpret large volumes of sensor data under time pressure. Current VLM applications include social media monitoring for situational awareness, generation of draft action plans, and translation of technical alerts into public-facing messages. While these efforts can accelerate information flow, they remain largely limited to decision-support roles. Such approaches can increase operator burden because humans must still translate outputs into coordinated actions across teams and robotic assets. This study explores the viability of embedding VLMs as coordination agents within the human-UAV loop. The proposed architecture integrates natural language interaction, mission-level task coordination, software-in-the-loop implementation, and communication aligned with the Incident Command System (ICS). Rather than functioning solely as advisory tools, VLMs facilitate communication between human operators, mission control logic, and UAV task execution. The framework was developed using a Model-Based Systems Engineering (MBSE) approach, with use case and block definition diagrams representing system roles, internal structure, and component interactions. Three key elements, the VLM Coordinator Agent, UAV Mission Control, and Task Allocator, were implemented within an integrated simulation and control environment. A preliminary human-factors evaluation with seven participants showed reduced perceived workload across mental demand, effort, and frustration, along with high ratings for AI trust and communication clarity. By integrating MBSE, software-in-the-loop testing, and human-factors evaluation, this work advances scalable human-autonomy teaming for high-stakes disaster response, with broader implications for aerospace autonomy and civil safety.

Related papers

• From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI [74.72113315839412]
  Large Language Models (LLMs) are undergoing a fundamental transformation into integrated AI systems capable of reasoning, memory, and self-improvement.<n>We conceptualize this transition as a shift from episodic to Digital Colleague: from conversational answers to persistent work.
  arXiv  Detail & Related papers  (2026-06-12T14:33:55Z)
• ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration [9.434833852972353]
  ROSClaw is an agent framework for heterogeneous robots that integrates policy learning and task execution within a unified vision-language model (VLM) controller.<n>By establishing an autonomous closedloop framework, ROSClaw minimizes the reliance on robot-specific development.
  arXiv  Detail & Related papers  (2026-04-06T13:16:24Z)
• Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses [168.50301366360344]
  Embodied Artificial Intelligence (Embodied AI) integrates perception, cognition, planning, and interaction into agents that operate in safety-critical environments.<n>Unlike digital AI systems, embodied agents must act under uncertain sensing, incomplete knowledge, and dynamic human-robot interactions.<n>This survey provides a comprehensive review of safety research in embodied AI, examining attacks and defenses across the full embodied pipeline.
  arXiv  Detail & Related papers  (2026-03-28T13:21:44Z)
• Reasoning Knowledge-Gap in Drone Planning via LLM-based Active Elicitation [15.477731207334793]
  Human-AI joint planning in Unmanned Aerial Vehicles (UAVs) typically relies on control handover when facing environmental uncertainties.<n>We propose a novel framework that shifts the collaboration paradigm from control takeover to active information elicitation.
  arXiv  Detail & Related papers  (2026-03-08T22:00:32Z)
• IMAGINE: Intelligent Multi-Agent Godot-based Indoor Networked Exploration [0.0]
  This paper implements Multi-Agent Reinforcement Learning (MARL) to address challenges in a 2D indoor environment.<n>Policy training aims to achieve emergent collaborative behaviours and decision-making under uncertainty.
  arXiv  Detail & Related papers  (2026-02-02T22:08:41Z)
• AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios [64.51320327698231]
  We introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios.<n>We develop an innovative semi-automated collaborative agent-based labeling assistant framework.<n>We also propose HawkEyeTrack, a novel method that collaboratively enhances vision-language representation learning.
  arXiv  Detail & Related papers  (2025-11-26T04:44:27Z)
• An LLM-based Framework for Human-Swarm Teaming Cognition in Disaster Search and Rescue [11.300720465575608]
  Large-scale disaster Search And Rescue (SAR) operations are persistently challenged by complex terrain and disrupted communications.<n>While Unmanned Aerial Vehicle (UAV) swarms offer a promising solution for tasks like wide-area search and supply delivery, yet their effective coordination places a significant cognitive burden on human operators.<n>This study proposes a novel LLM-CRF system that leverages Large Language Models (LLMs) to model and augment human-swarm teaming cognition.
  arXiv  Detail & Related papers  (2025-11-06T04:27:20Z)
• AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds [12.464941027105306]
  AI for IT Operations (AIOps) aims to automate complex operational tasks, such as fault localization and root cause analysis, to reduce human workload and minimize customer impact.<n>Recent advances in Large Language Models (LLMs) and AI agents are revolutionizing AIOps by enabling end-to-end and multitask automation.<n>We present AIOPSLAB, a framework that deploys microservice cloud environments, injects faults, generates workloads, and exports telemetry data but also orchestrates these components and provides interfaces for interacting with and evaluating agents.
  arXiv  Detail & Related papers  (2025-01-12T04:17:39Z)
• Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics [68.36528819227641]
  This paper systematically evaluates the robustness of Vision-Language-Action (VLA) models.<n>We introduce two untargeted attack objectives that leverage spatial foundations to destabilize robotic actions, and a targeted attack objective that manipulates the robotic trajectory.<n>We design an adversarial patch generation approach that places a small, colorful patch within the camera's view, effectively executing the attack in both digital and physical environments.
  arXiv  Detail & Related papers  (2024-11-18T01:52:20Z)
• Compromising Embodied Agents with Contextual Backdoor Attacks [69.71630408822767]
  Large language models (LLMs) have transformed the development of embodied intelligence. This paper uncovers a significant backdoor security threat within this process. By poisoning just a few contextual demonstrations, attackers can covertly compromise the contextual environment of a black-box LLM.
  arXiv  Detail & Related papers  (2024-08-06T01:20:12Z)
• Unified Human-Scene Interaction via Prompted Chain-of-Contacts [61.87652569413429]
  Human-Scene Interaction (HSI) is a vital component of fields like embodied AI and virtual reality. This paper presents a unified HSI framework, UniHSI, which supports unified control of diverse interactions through language commands.
  arXiv  Detail & Related papers  (2023-09-14T17:59:49Z)
• Chat with the Environment: Interactive Multimodal Perception Using Large Language Models [19.623070762485494]
  Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment.
  arXiv  Detail & Related papers  (2023-03-14T23:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.