SkySim: A ROS2-based Simulation Environment for Natural Language Control of Drone Swarms using Large Language Models

• URL: http://arxiv.org/abs/2602.01226v1
• Date: Sun, 01 Feb 2026 13:34:34 GMT
• Title: SkySim: A ROS2-based Simulation Environment for Natural Language Control of Drone Swarms using Large Language Models
• Abstract summary: Unmanned Aerial Vehicle (UAV) swarms offer versatile applications in logistics, agriculture, and surveillance.<n>Traditionally static methods limit adaptability, while Large Language Models (LLMs) enable natural language control but generate unsafe trajectories.<n>This paper introduces SkySim, a ROS2-based simulation framework in Gazebo that decouples LLM high-level planning from low-level safety enforcement.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unmanned Aerial Vehicle (UAV) swarms offer versatile applications in logistics, agriculture, and surveillance, yet controlling them requires expert knowledge for safety and feasibility. Traditional static methods limit adaptability, while Large Language Models (LLMs) enable natural language control but generate unsafe trajectories due to lacking physical grounding. This paper introduces SkySim, a ROS2-based simulation framework in Gazebo that decouples LLM high-level planning from low-level safety enforcement. Using Gemini 3.5 Pro, SkySim translates user commands (e.g., "Form a circle") into spatial waypoints, informed by real-time drone states. An Artificial Potential Field (APF) safety filter applies minimal adjustments for collision avoidance, kinematic limits, and geo-fencing, ensuring feasible execution at 20 Hz. Experiments with swarms of 3, 10, and 30 Crazyflie drones validate spatial reasoning accuracy (100% across tested geometric primitives), real-time collision prevention, and scalability. SkySim empowers non-experts to iteratively refine behaviors, bridging AI cognition with robotic safety for dynamic environments. Future work targets hardware integration.

Related papers

• DESCENT: Directed Edge Scene Encoding for Airport Surface Movement Prediction [54.94692733670454]
  DESCENT is a transformer-based architecture designed to handle heterogeneous dynamics and strict topological constraints.<n>Our approach features a Potential Reachable Set (PRS) context sampling mechanism that adaptively collects airfield environment context.<n>Our evaluations on the Amelia-10 benchmark demonstrate significant performance improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-26T16:49:50Z)
• Generative AI for Safe and Photorealistic Drone Light Shows [51.71264054256171]
  Drone light shows are redefining aerial entertainment, yet their widespread adoption is bottlenecked by labor-intensive, manual animation.<n>We introduce SWAN, an end-to-end pipeline that synthesizes photorealistic, large-scale, and collision-free drone choreographies directly from text prompts.<n>We demonstrate scalability by safely orchestrating simulated 2,000-drone formations and validate physical feasibility on a dense real-world swarm of 49 quadcopters.
  arXiv  Detail & Related papers  (2026-06-24T06:34:44Z)
• Simulation-Informed Diffusion for Decentralized Multi-robot Motion Planning [56.240199425429445]
  Multi-robot motion planning requires each robot to generate collision-free trajectories from local observations.<n>Most existing planners generate trajectories from a static snapshot of the local observation.<n>This paper introduces Simulation-Informed Diffusion (SID), a decentralized framework built on constraint-aware diffusion models.
  arXiv  Detail & Related papers  (2026-05-26T21:17:53Z)
• Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls [57.166020875486474]
  We propose textbfAero-World, a method for converting a pretrained image-to-video diffusion model into a controllable aerial video generator.<n>A frozen latent-space Physics Probe, trained independently on real video--IMU pairs, provides differentiable inertial-consistency supervision during LoRA finetuning.<n>On AeroBench, Aero-World improves mean AAS from 57.7 to 63.6 over action-only finetuning and gives a stronger quality-control trade-off than AirScape.
  arXiv  Detail & Related papers  (2026-05-19T12:02:17Z)
• Learning Reactive Dexterous Grasping via Hierarchical Task-Space RL Planning and Joint-Space QP Control [50.28263951510334]
  We propose a hybrid hierarchical control framework for reactive dexterous grasping.<n>The proposed framework explicitly decouples high-level spatial intent from low-level joint execution.<n>We extensively validate the proposed framework through a rigorous simulation-to-reality pipeline.
  arXiv  Detail & Related papers  (2026-05-05T04:49:38Z)
• MARS-Dragonfly: Agile and Robust Flight Control of Modular Aerial Robot Systems [21.471271822760276]
  Modular Aerial Robot Systems (MARS) comprise multiple drone units with reconfigurable connected formations.<n>MARS rely on simplified quasi-static models and rule-based allocation, which generate discontinuous and unbounded motor commands.<n>We develop a compact mechanical system that enables passive docking, detection-free passive locking, and magnetic-assisted separation.
  arXiv  Detail & Related papers  (2026-04-07T06:43:08Z)
• SafeLand: Safe Autonomous Landing in Unknown Environments with Bayesian Semantic Mapping [41.90149506430796]
  We introduce SafeLand, a vision-based system for safe autonomous landing (SAL)<n>Our approach constructs an online semantic ground map via deep learning-based semantic segmentation, optimized for embedded deployment.<n>We extensively evaluate our method in 200 simulations and 60 end-to-end field tests across industrial, urban, and rural environments at altitudes up to 100m.
  arXiv  Detail & Related papers  (2026-03-18T07:11:54Z)
• NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environments [56.35569661650558]
  We introduce NOVA, a fully onboard, object-centric framework that enables robust target tracking and collision-aware navigation.<n>Rather than constructing a global map, NOVA formulates perception, estimation, and control entirely in the target's reference frame.<n>We validate NOVA across challenging real-world scenarios, including urban mazes, forest trails, and repeated transitions through buildings with intermittent GPS loss.
  arXiv  Detail & Related papers  (2025-06-23T14:28:30Z)
• AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions [64.85086226439954]
  We present SAFE, a benchmark for assessing the safety of embodied VLM agents on hazardous instructions.<n> SAFE comprises three components: SAFE-THOR, SAFE-VERSE, and SAFE-DIAGNOSE.<n>We uncover systematic failures in translating hazard recognition into safe planning and execution.
  arXiv  Detail & Related papers  (2025-06-17T16:37:35Z)
• Low-altitude UAV Friendly-Jamming for Satellite-Maritime Communications via Generative AI-enabled Deep Reinforcement Learning [72.23178920029957]
  This paper presents a satellite-maritime communication system assisted by low-altitude unmanned aerial vehicle (UAV) friendly-jamming.<n>We formulate a secure satellite-maritime communication multi-objective optimization problem (SSMCMOP)<n>In order to solve the dynamic and long-term optimization problem, we reformulate it into a Markov decision process.<n>We then propose a transformer-enhanced soft actor-critic (TransSAC) algorithm, which is a generative artificial intelligence-enabled deep reinforcement learning approach.
  arXiv  Detail & Related papers  (2025-01-26T10:13:51Z)
• ASMA: An Adaptive Safety Margin Algorithm for Vision-Language Drone Navigation via Scene-Aware Control Barrier Functions [9.645098673995317]
  We consider a VLN-operated drone platform and enhance its safety by formulating a novel scene-aware CBF.<n>A CBF-less baseline system uses a Vision-Language with cross-modal attention to convert commands into an ordered sequence of landmarks.<n>ASMA tracks moving objects and performs scene-aware CBF evaluation on-the-fly, which serves as an additional constraint.
  arXiv  Detail & Related papers  (2024-09-16T13:44:50Z)
• ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable [88.08120417169971]
  Machine learning based autonomous driving systems often face challenges with safety-critical scenarios that are rare in real-world data. This work explores generating safety-critical driving scenarios by modifying complex real-world regular scenarios through trajectory optimization. Our approach addresses unrealistic diverging trajectories and unavoidable collision scenarios that are not useful for training robust planner.
  arXiv  Detail & Related papers  (2024-09-12T08:26:33Z)
• You Only Crash Once: Improved Object Detection for Real-Time, Sim-to-Real Hazardous Terrain Detection and Classification for Autonomous Planetary Landings [7.201292864036088]
  A cheap and effective way of detecting hazardous terrain is through the use of visual cameras. Traditional techniques for visual hazardous terrain detection focus on template matching and registration to pre-built hazard maps. We introduce You Only Crash Once (YOCO), a deep learning-based visual hazardous terrain detection and classification technique.
  arXiv  Detail & Related papers  (2023-03-08T21:11:51Z)
• Safe Reinforcement Learning Using Black-Box Reachability Analysis [20.875010584486812]
  Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. We propose a Black-box Reachability-based Safety Layer (BRSL) with three main components.
  arXiv  Detail & Related papers  (2022-04-15T10:51:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.