CosFly: Plan in the Matrix, Fly in the World

• URL: http://arxiv.org/abs/2605.19120v1
• Date: Mon, 18 May 2026 21:11:18 GMT
• Title: CosFly: Plan in the Matrix, Fly in the World
• Abstract summary: We present CosFly, a box-structured planning and multimodal simulation pipeline for aerial tracking.<n>We also present CosFly-Track, a large-scale UAV dataset for dynamic target tracking across diverse environments.
• Score: 5.765003525486641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present CosFly, a box-structured planning and multimodal simulation pipeline for aerial tracking, together with CosFly-Track, a large-scale UAV dataset for dynamic target tracking across diverse environments including urban centers, highways, rural landscapes, forests, and coastal towns. In our current implementation on CARLA, CosFly provides a modular 7-step construction pipeline that converts complex 3D worlds into structured obstacle representations for planning, then projects the resulting trajectories back into multi-modal sensor data -- including RGB images, high-precision depth maps, and semantic segmentation masks -- paired with natural language navigation instructions. A key feature is the support for configurable fixed-FOV zoom levels (one FOV setting drawn per trajectory and held constant throughout), enabling simulation of various focal lengths through camera-intrinsic adjustments. The pipeline covers the complete workflow from 3D map export through grid simplification, pedestrian and drone trajectory planning, multi-modal rendering with 6-DOF pose annotations, quality inspection, and teacher-student caption generation. We analyze two trajectory-planning paradigms for aerial target tracking: a conventional two-stage pipeline with front-end candidate generation and backend refinement, and a direct gradient-based formulation that optimizes multiple tracking constraints in a single objective. The public CosFly-Track release contains 250 validated trajectories and approximately 100,000 rendered images with complete 6-DOF drone pose annotations (position x, y, z and orientation yaw, pitch, roll). Together, the pipeline and dataset establish a scalable foundation for aerial-ground collaborative research, supporting dynamic target tracking, UAV navigation, and multi-modal perception across diverse environments.

Related papers

• FlyMirage: A Fully Automated Generation Pipeline for Diverse and Scalable UAV Flight Data via Generative World Model [13.125579202955384]
  FlyMirage is a highly scalable and fully automated data generation pipeline for aerial VLN.<n>We generate a large-scale, diverse, and photorealistic aerial VLN dataset, with dynamically feasible flying trajectories.
  arXiv  Detail & Related papers  (2026-05-19T09:41:04Z)
• CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization [5.936588226093352]
  We introduce CosFlyTrack, a large-scale multi-modal dataset for UAV visual tracking in urban environments.<n>The dataset provides approximately 12,000 expert and perturbed UAV trajectories generated from 6,000 pedestrian paths.<n>CosFlyTrack improves tracking performance to 78.3 to 95.6% SR@1 meter, a 53 to 69 percentage point gain over zero-shot baselines.
  arXiv  Detail & Related papers  (2026-05-18T02:49:58Z)
• S3KF: Spherical State-Space Kalman Filtering for Panoramic 3D Multi-Object Tracking [24.79542959759503]
  Panoramic multi-object tracking is important for industrial safety monitoring, wide-area robotic perception, and infrastructure-light deployment in large workspaces.<n>Existing image-plane trackers are tightly coupled to the camera projection and become unreliable in panoramic imagery.<n>We present a panoramic 3D multi-object tracking framework built on a motorized rotating LiDAR and a quad-fisheye camera rig.
  arXiv  Detail & Related papers  (2026-03-29T06:01:29Z)
• TrajVG: 3D Trajectory-Coupled Visual Geometry Learning [34.47385438436523]
  TrajVG is a reconstruction framework that makes cross-frame 3D correspondence explicit.<n>We reformulate the same coupling constraints into self-supervised objectives using only pseudo 2D tracks.<n>Experiments across 3D tracking, pose estimation, pointmap reconstruction, and video depth show that TrajVG surpasses the current feedforward performance baseline.
  arXiv  Detail & Related papers  (2026-02-04T11:13:55Z)
• PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-Forward Planar Splatting [56.188624157291024]
  We introduce PLANA3R, a pose-free framework for metric Planar 3D Reconstruction from unposed two-view images.<n>Unlike prior feedforward methods that require 3D plane annotations during training, PLANA3R learns planar 3D structures without explicit plane supervision.<n>We validate PLANA3R on multiple indoor-scene datasets with metric supervision and demonstrate strong generalization to out-of-domain indoor environments.
  arXiv  Detail & Related papers  (2025-10-21T15:15:33Z)
• MapAnything: Universal Feed-Forward Metric 3D Reconstruction [63.79151976126576]
  MapAnything ingests one or more images along with optional geometric inputs such as camera intrinsics, poses, depth, or partial reconstructions.<n>It then directly regresses the metric 3D scene geometry and cameras.<n>MapAnything addresses a broad range of 3D vision tasks in a single feed-forward pass.
  arXiv  Detail & Related papers  (2025-09-16T18:00:14Z)
• SpatialTrackerV2: 3D Point Tracking Made Easy [73.0350898700048]
  SpatialTrackerV2 is a feed-forward 3D point tracking method for monocular videos.<n>It decomposes world-space 3D motion into scene geometry, camera ego-motion, and pixel-wise object motion.<n>By learning geometry and motion jointly from such heterogeneous data, SpatialTrackerV2 outperforms existing 3D tracking methods by 30%.
  arXiv  Detail & Related papers  (2025-07-16T17:59:03Z)
• MuRF: Multi-Baseline Radiance Fields [117.55811938988256]
  We present Multi-Baseline Radiance Fields (MuRF), a feed-forward approach to solving sparse view synthesis. MuRF achieves state-of-the-art performance across multiple different baseline settings. We also show promising zero-shot generalization abilities on the Mip-NeRF 360 dataset.
  arXiv  Detail & Related papers  (2023-12-07T18:59:56Z)
• Vision-based Large-scale 3D Semantic Mapping for Autonomous Driving Applications [53.553924052102126]
  We present a complete pipeline for 3D semantic mapping solely based on a stereo camera system. The pipeline comprises a direct visual odometry front-end as well as a back-end for global temporal integration. We propose a simple but effective voting scheme which improves the quality and consistency of the 3D point labels.
  arXiv  Detail & Related papers  (2022-03-02T13:18:38Z)
• Monocular Road Planar Parallax Estimation [25.36368935789501]
  Estimating the 3D structure of the drivable surface and surrounding environment is a crucial task for assisted and autonomous driving. We propose Road Planar Parallax Attention Network (RPANet), a new deep neural network for 3D sensing from monocular image sequences. RPANet takes a pair of images aligned by the homography of the road plane as input and outputs a $gamma$ map for 3D reconstruction.
  arXiv  Detail & Related papers  (2021-11-22T10:03:41Z)
• OmniSLAM: Omnidirectional Localization and Dense Mapping for Wide-baseline Multi-camera Systems [88.41004332322788]
  We present an omnidirectional localization and dense mapping system for a wide-baseline multiview stereo setup with ultra-wide field-of-view (FOV) fisheye cameras. For more practical and accurate reconstruction, we first introduce improved and light-weighted deep neural networks for the omnidirectional depth estimation. We integrate our omnidirectional depth estimates into the visual odometry (VO) and add a loop closing module for global consistency.
  arXiv  Detail & Related papers  (2020-03-18T05:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.