A Universal Large Language Model -- Drone Command and Control Interface

• URL: http://arxiv.org/abs/2601.15486v1
• Date: Wed, 21 Jan 2026 21:37:54 GMT
• Title: A Universal Large Language Model -- Drone Command and Control Interface
• Abstract summary: We develop and deploy a cloud based Linux machine hosting an MCP server that supports the Mavlink protocol.<n>We demonstrate flight control of a real unmanned aerial vehicle.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of artificial intelligence (AI) for drone control can have a transformative impact on drone capabilities, especially when real world information can be integrated with drone sensing, command, and control, part of a growing field of physical AI. Large language models (LLMs) can be advantageous if trained at scale on general knowledge, but especially and in particular when the training data includes information such as detailed map geography topology of the entire planet, as well as the ability to access real time situational data such as weather. However, challenges remain in the interface between drones and LLMs in general, with each application requiring a tedious, labor intensive effort to connect the LLM trained knowledge to drone command and control. Here, we solve that problem, using an interface strategy that is LLM agnostic and drone agnostic, providing the first universal, versatile, comprehensive and easy to use drone control interface. We do this using the new model context protocol (MCP) standard, an open standard that provides a universal way for AI systems to access external data, tools, and services. We develop and deploy a cloud based Linux machine hosting an MCP server that supports the Mavlink protocol, an ubiquitous drone control language used almost universally by millions of drones including Ardupilot and PX4 framework.We demonstrate flight control of a real unmanned aerial vehicle. In further testing, we demonstrate extensive flight planning and control capability in a simulated drone, integrated with a Google Maps MCP server for up to date, real time navigation information. This demonstrates a universal approach to integration of LLMs with drone command and control, a paradigm that leverages and exploits virtually all of modern AI industry with drone technology in an easy to use interface that translates natural language to drone control.

Related papers

• Show-Harness: Just a VLM Agent Can Play Robots [53.80887083886227]
  Embodied vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging.<n>We present Show-Harness, an Embodied Harness that enablesVLMs to "play" robots through a compact semantic interface linking intent to action.<n>We further develop GUMI, which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware.
  arXiv  Detail & Related papers  (2026-09-09T17:53:38Z)
• Real-Time Visual Intelligence on Low-Cost UAVs: A Modular Approach for Tracking, Scanning, and Navigation [0.0]
  This paper presents the development of an integrated intelligent drone system designed to serve as a personal assistant.<n>We implement a modular architecture that integrates three core artificial intelligence functionalities: facial detection, facial recognition, and depth estimation from monocular vision.<n>Unlike existing commercial solutions, this system emphasizes accessibility, low-cost hardware, and open-source technologies.
  arXiv  Detail & Related papers  (2026-07-02T15:15:56Z)
• AeroGen: Agentic Drone Autonomy through Single-Shot Structured Prompting & Drone SDK [3.1941554288428193]
  AeroGen is an open-loop framework that enables consistently correct single-shot AI-generated drone control programs.<n>We show that AeroGen generates about 40 lines of AeroD Python code in about 20s per mission, in both real-world and simulations.
  arXiv  Detail & Related papers  (2026-03-15T06:16:02Z)
• Large Language Models to Enhance Multi-task Drone Operations in Simulated Environments [7.546510363080917]
  We propose a method that integrates a fine-tuned CodeT5 model with the Unreal Engine-based AirSim drone simulator.<n>The proposed method exhibits superior task execution efficiency and command understanding capabilities in simulated environments.
  arXiv  Detail & Related papers  (2026-01-13T10:21:17Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• Chasing the Intruder: A Reinforcement Learning Approach for Tracking Intruder Drones [0.08192907805418582]
  We propose a reinforcement learning based approach for identifying and tracking any intruder drone using a chaser drone. Our proposed solution uses computer vision techniques interleaved with the policy learning framework of reinforcement learning. The results show that the reinforcement learning based policy converges to identify and track the intruder drone.
  arXiv  Detail & Related papers  (2023-09-10T16:31:40Z)
• Multi-model fusion for Aerial Vision and Dialog Navigation based on human attention aids [69.98258892165767]
  We present an aerial navigation task for the 2023 ICCV Conversation History. We propose an effective method of fusion training of Human Attention Aided Transformer model (HAA-Transformer) and Human Attention Aided LSTM (HAA-LSTM) models.
  arXiv  Detail & Related papers  (2023-08-27T10:32:52Z)
• TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone Detection in Aerial Videos [57.92385818430939]
  Drone-to-drone detection using visual feed has crucial applications, such as detecting drone collisions, detecting drone attacks, or coordinating flight with other drones. Existing methods are computationally costly, follow non-end-to-end optimization, and have complex multi-stage pipelines, making them less suitable for real-time deployment on edge devices. We propose a simple yet effective framework, itTransVisDrone, that provides an end-to-end solution with higher computational efficiency.
  arXiv  Detail & Related papers  (2022-10-16T03:05:13Z)
• GNM: A General Navigation Model to Drive Any Robot [67.40225397212717]
  General goal-conditioned model for vision-based navigation can be trained on data obtained from many distinct but structurally similar robots. We analyze the necessary design decisions for effective data sharing across robots. We deploy the trained GNM on a range of new robots, including an under quadrotor.
  arXiv  Detail & Related papers  (2022-10-07T07:26:41Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)
• Aerial Vision-and-Dialog Navigation [10.596163697911525]
  We introduce Aerial Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language conversation. We build a drone simulator with a continuous environment and collect a new AVDN dataset of over 3k recorded navigation trajectories. We propose an effective Human Attention Aided Transformer model (HAA-Transformer) which learns to predict both navigation waypoints and human attention.
  arXiv  Detail & Related papers  (2022-05-24T17:28:14Z)
• Learn by Observation: Imitation Learning for Drone Patrolling from Videos of A Human Navigator [22.06785798356346]
  We propose to let the drone learn patrolling in the air by observing and imitating how a human navigator does it on the ground. The observation process enables the automatic collection and annotation of data using inter-frame geometric consistency. A newly designed neural network is trained based on the annotated data to predict appropriate directions and translations.
  arXiv  Detail & Related papers  (2020-08-30T15:20:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.