AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight

• URL: http://arxiv.org/abs/2607.14997v1
• Date: Thu, 16 Jul 2026 13:46:00 GMT
• Title: AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight
• Abstract summary: Language-conditioned quadrotor flight requires a policy to ground semantic goals.<n>We present AeroAct, an action-centered world-action model (WAM) for quadrotor navigation.
• Score: 42.75555298269899
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language-conditioned quadrotor flight requires a policy to ground semantic goals, anticipate the visual consequences of ego-motion, and output control references that remain smooth and dynamically executable under rapidly changing first-person views. Existing aerial vision-language navigation and vision-language-action methods commonly use discrete actions, high-level waypoints, or instantaneous velocity commands, which provide limited supervision about how flight actions change future observations. We present AeroAct, an action-centered world-action model (WAM) for quadrotor navigation. To the best of our knowledge, AeroAct is the first WAM instantiated and demonstrated for real-world aerial flight. The model adapts a pretrained video diffusion Transformer to predict local trajectory-action chunks from egocentric visual history, proprioception, and language. Future first-person frames are used during training as dense consequence supervision, while deployment directly decodes actions without generating future video. To obtain aligned visual, state, language, and dynamically feasible action data, we build a DiffAero-based pipeline with complementary Isaac Lab and 3D Gaussian splatting renderers. We further introduce a low-cost handheld collection device that couples camera observations with motion estimates to recreate flight-like egocentric trajectories, and a self-guidance procedure that improves temporal consistency across overlapping trajectory chunks. Closed-loop simulation and real-world experiments show that temporal visual context improves target tracking and object-search performance, and that WAM-based policies can be executed on a physical quadrotor.

Related papers

• World Pilot: Steering Vision-Language-Action Models with World-Action Priors [21.867210989881016]
  Vision-Language-Action (VLA) models inherit semantic grounding from large-scale pretraining.<n>World Pilot augments the policy with priors from a World-Action Model (WAM)<n>World Pilot attains a state-of-the-art Total success rate of 84.7% on the LIBERO-Plus zero-shot OOD benchmark.
  arXiv  Detail & Related papers  (2026-06-10T17:59:08Z)
• ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning [8.298446181889675]
  Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability.<n>We propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling.<n>ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights.
  arXiv  Detail & Related papers  (2026-05-31T12:39:44Z)
• Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls [57.166020875486474]
  We propose textbfAero-World, a method for converting a pretrained image-to-video diffusion model into a controllable aerial video generator.<n>A frozen latent-space Physics Probe, trained independently on real video--IMU pairs, provides differentiable inertial-consistency supervision during LoRA finetuning.<n>On AeroBench, Aero-World improves mean AAS from 57.7 to 63.6 over action-only finetuning and gives a stronger quality-control trade-off than AirScape.
  arXiv  Detail & Related papers  (2026-05-19T12:02:17Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• From Time Series to State: Situation-Aware Modeling for Air Traffic Flow Prediction [28.381858327505828]
  We propose textitAeroSense, a direct state-to-flow modeling framework for air traffic prediction.<n>AeroSense represents the real-time airspace situation as textita dynamic set of aircraft states, enabling the direct processing of a variable number of aircraft.<n>Experiments on a large-scale real-world airport dataset demonstrate that AeroSense consistently achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-13T08:54:19Z)
• AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps [7.710034405765985]
  AIM is an intent-aware unified world action model that bridges this gap via an explicit spatial interface.<n>Built on a pretrained video generation model, AIM jointly models future observations and value maps within a shared mixture-of-transformers architecture.<n>Experiments on RoboTwin 2.0 benchmark show that AIM achieves a 94.0% average success rate, significantly outperforming prior unified world action baselines.
  arXiv  Detail & Related papers  (2026-04-13T07:48:58Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• Open-World Drone Active Tracking with Goal-Centered Rewards [62.21394499788672]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations.<n>We propose DAT, the first open-world drone active air-to-ground tracking benchmark.<n>We also propose GC-VAT, which aims to improve the performance of drone tracking targets in complex scenarios.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion [83.88829943619656]
  We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. Our guided diffusion model allows users to constrain trajectories through target waypoints, speed, and specified social groups. We propose utilizing the value function learned during RL training of the animation controller to guide diffusion to produce trajectories better suited for particular scenarios.
  arXiv  Detail & Related papers  (2023-04-04T15:46:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.