FlowPilot: Real-Time World-Action Modeling for Agile UAV Navigation

• URL: http://arxiv.org/abs/2608.00635v1
• Date: Sat, 01 Aug 2026 12:43:31 GMT
• Title: FlowPilot: Real-Time World-Action Modeling for Agile UAV Navigation
• Abstract summary: FlowPilot is a compact world-action model for real-time onboard UAV navigation from depth.<n>A dual-stream mixture-of-transformers couples video and action experts through shared attention, allowing future-scene prediction and trajectory generation to inform each other.
• Score: 18.092834753587454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present FlowPilot, a compact world-action model for real-time onboard UAV navigation from depth. Unlike map-then-optimize pipelines that require local reconstruction or end-to-end policies that lack explicit scene prediction, FlowPilot jointly denoises future depth observations and executable trajectories with flow matching. A dual-stream mixture-of-transformers couples video and action experts through shared attention, allowing future-scene prediction and trajectory generation to inform each other. At deployment, the model runs action-centrically and outputs only a trajectory. To ensure trackability, actions are parameterized as degree-7 Bernstein polynomials: the current state constrains the initial control points, and the network predicts five free control points, yielding C^2-continuous references with closed-form velocity, acceleration and jerk. FlowPilot is trained on a three-level depth pyramid spanning high-throughput simulation, photorealistic simulation, and real onboard data. In closed-loop simulation, it outperforms learning- and optimization-based baselines under increasing clutter and commanded speeds up to 8m/s. On a physical quadrotor, the full perception-to-action pipeline runs in under 18ms on a Jetson Orin NX and reaches 5.5m/s in cluttered indoor and forest environments using only onboard sensing and computation.

Related papers

• MAD: Mapping-Aware World Models for Agile Quadrotor Flight [47.458890396048126]
  Mapping-Aware Dreamer (MAD) is a geometry-aware world model for vision-based quadrotor flight.<n>MAD learns recurrent latent dynamics that reconstruct robocentric occupancy and visibility grid maps.<n>We demonstrate safe indoor and outdoor flight under limited sensing, reaching 9.66 m/s in simulation and 5.05 m/s in real-world forest experiments.
  arXiv  Detail & Related papers  (2026-06-03T07:17:57Z)
• RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation [29.85979864354417]
  Planning and acting in 3D environments is a fundamental capability for robotic manipulation in the real world.<n>We introduce RoboFlow4D, a lightweight flow world model that unifies perception and planning by estimating temporal motion in physical 3D space.
  arXiv  Detail & Related papers  (2026-05-17T16:11:22Z)
• FlowAD: Ego-Scene Interactive Modeling for Autonomous Driving [20.991905328262746]
  We introduce a novel ego-scene interactive modeling paradigm.<n>Inspired by human recognition, the paradigm represents ego-scene interaction as the scene flow relative to the ego-vehicle.<n>Within it, an ego-guided scene partition first constructs basic flow units to quantify scene flow.<n>Then, based on flow units, spatial and temporal flow predictions are performed to model dynamics of scene flow.
  arXiv  Detail & Related papers  (2026-03-11T16:28:53Z)
• CoFL: Continuous Flow Fields for Language-Conditioned Navigation [14.35468089349405]
  CoFL is an end-to-end policy that maps a bird's-eye view (BEV) observation and a language instruction to a continuous flow field for navigation.<n>By training on a mixed distribution, CoFL significantly outperforms modular Vision-Language Model (VLM)-based planners.
  arXiv  Detail & Related papers  (2026-03-03T11:02:55Z)
• Trace Anything: Representing Any Video in 4D via Trajectory Fields [98.85848134960172]
  Trajectory Field is a dense mapping that assigns a continuous 3D trajectory function of time to each pixel in every frame.<n>We introduce Trace Anything, a neural network that predicts the entire trajectory field in a single feed-forward pass.<n>We trained the Trace Anything model on large-scale 4D data, including data from our new platform.
  arXiv  Detail & Related papers  (2025-10-15T17:59:04Z)
• SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving [5.343552118560704]
  SpaRC-AD is a query-based end-to-end camera-radar fusion framework for planning-oriented autonomous driving.<n>Our method achieves strong improvements over the state-of-the-art vision-only baselines across multiple autonomous driving tasks.
  arXiv  Detail & Related papers  (2025-08-14T12:02:41Z)
• m4: A Learned Flow-level Network Simulator [14.802940782072122]
  Flow-level simulators abstract traffic as continuous flows with dynamically assigned transmission rates.<n>We present m4, an accurate and scalable flow-level simulator that uses machine learning to learn the dynamics of the network of interest.
  arXiv  Detail & Related papers  (2025-03-03T17:51:08Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion [83.88829943619656]
  We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. Our guided diffusion model allows users to constrain trajectories through target waypoints, speed, and specified social groups. We propose utilizing the value function learned during RL training of the animation controller to guide diffusion to produce trajectories better suited for particular scenarios.
  arXiv  Detail & Related papers  (2023-04-04T15:46:42Z)
• StreamYOLO: Real-time Object Detection for Streaming Perception [84.2559631820007]
  We endow the models with the capacity of predicting the future, significantly improving the results for streaming perception. We consider multiple velocities driving scene and propose Velocity-awared streaming AP (VsAP) to jointly evaluate the accuracy. Our simple method achieves the state-of-the-art performance on Argoverse-HD dataset and improves the sAP and VsAP by 4.7% and 8.2% respectively.
  arXiv  Detail & Related papers  (2022-07-21T12:03:02Z)
• Real-time Object Detection for Streaming Perception [84.2559631820007]
  Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. We build a simple and effective framework for streaming perception. Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
  arXiv  Detail & Related papers  (2022-03-23T11:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.