CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization

• URL: http://arxiv.org/abs/2605.17776v2
• Date: Wed, 20 May 2026 07:06:32 GMT
• Title: CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization
• Abstract summary: We introduce CosFlyTrack, a large-scale multi-modal dataset for UAV visual tracking in urban environments.<n>The dataset provides approximately 12,000 expert and perturbed UAV trajectories generated from 6,000 pedestrian paths.<n>CosFlyTrack improves tracking performance to 78.3 to 95.6% SR@1 meter, a 53 to 69 percentage point gain over zero-shot baselines.
• Score: 5.936588226093352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent aerial vision-language navigation (VLN) datasets have grown rapidly, but they primarily address goal-oriented navigation to static destinations, leaving UAV visual tracking -- continuously following a moving target while maintaining visibility -- largely without dedicated training data. We introduce CosFlyTrack, a large-scale multi-modal dataset and scalable generation pipeline for UAV visual tracking in urban environments. The dataset provides approximately 12,000 expert and perturbed UAV trajectories generated from 6,000 pedestrian paths, comprising 2.4 million timesteps (approximately 334 hours) with seven aligned data channels: RGB, metric depth, semantic segmentation, six-degree-of-freedom drone pose, target state with visibility flag, bilingual (Chinese-English) instructions, and trajectory-pair metadata. To generate high-quality expert trajectories, we develop MuCO, a multi-constraint optimizer that plans directly in continuous three-dimensional space with BVH-accelerated collision and visibility queries, jointly enforcing target visibility, viewpoint quality, collision avoidance, smoothness, and kinematic feasibility, avoiding the discretization artifacts and post-hoc smoothing of grid-based planners. Fine-tuning experiments on seven vision-language models show that CosFlyTrack improves tracking performance to 78.3 to 95.6 percent SR@1 meter, a 53 to 69 percentage point gain over zero-shot baselines, supporting the dataset as a training resource for dynamic target-following agents. The dataset is publicly available at https://huggingface.co/datasets/AutelRobotics/CosFly; evaluation scripts and pre-trained checkpoints are hosted at https://huggingface.co/AutelRobotics/CosFly-Track.

Related papers

• RRTrack: Robust and Recoverable Object 6D Pose Tracking for Dynamic Scenes [55.15267980622371]
  RRTrack is an efficient, recoverable object 6D pose tracker.<n>It enables robust tracking through fast motion and target disappearance--reappearance.<n>Experiments show RRTrack improves equal-subset mean ADD-S AR by 66.3% and ADD-S AUC by 65.7% over FoundationPose.
  arXiv  Detail & Related papers  (2026-07-26T14:05:46Z)
• CosFly: Plan in the Matrix, Fly in the World [5.765003525486641]
  We present CosFly, a box-structured planning and multimodal simulation pipeline for aerial tracking.<n>We also present CosFly-Track, a large-scale UAV dataset for dynamic target tracking across diverse environments.
  arXiv  Detail & Related papers  (2026-05-18T21:11:18Z)
• Track A*: Fast Visibility-Aware Trajectory Planning for Active Target Tracking [1.8297494098768168]
  We present Track A star (TA star), an offline search-based trajectory planner for visibility-aware target tracking.<n>TA star combines a layered Acyclic Graph (D) search with three engineering optimizations: cross-time obstacle distance caching against a Bounding Volume, per-layer beam pruning, and a multi-ray visibility evaluator.<n>Our empirical results demonstrate robust, near-baseline visibility performance at a fraction of the computational cost.
  arXiv  Detail & Related papers  (2026-05-06T18:09:15Z)
• UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models [14.433739259355406]
  Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks.<n>To benchmark multimodal tracking in such environments, we construct a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects.<n>We propose an improved VLA tracking model, UAV-Track VLA, built upon the $_0.5$ architecture.
  arXiv  Detail & Related papers  (2026-04-02T16:33:38Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• Learning to Track Any Points from Human Motion [55.831218129679144]
  We propose an automated pipeline to generate pseudo-labeled training data for point tracking.<n>A point tracking model trained on AnthroTAP achieves annotated state-of-the-art performance on the TAP-Vid benchmark.
  arXiv  Detail & Related papers  (2025-07-08T17:59:58Z)
• MUST: The First Dataset and Unified Framework for Multispectral UAV Single Object Tracking [17.96400810834486]
  We introduce the first large-scale Multispectral UAV Single Object Tracking dataset (MUST)<n>MUST includes 250 video sequences spanning diverse environments and challenges.<n>We also propose a novel tracking framework, UNTrack, which encodes unified spectral, spatial, and temporal features from spectrum prompts.
  arXiv  Detail & Related papers  (2025-03-22T08:47:28Z)
• UAVD4L: A Large-Scale Dataset for UAV 6-DoF Localization [14.87295056434887]
  We introduce a large-scale 6-DoF UAV dataset for localization (UAVD4L) We develop a two-stage 6-DoF localization pipeline (UAVLoc), which consists of offline synthetic data generation and online visual localization. Results on the new dataset demonstrate the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2024-01-11T15:19:21Z)
• BEVTrack: A Simple and Strong Baseline for 3D Single Object Tracking in Bird's-Eye View [54.48052449493636]
  3D Single Object Tracking (SOT) is a fundamental task in computer vision and plays a critical role in applications like autonomous driving.<n>We propose BEVTrack, a simple yet effective motion-based tracking method.<n>We show that BEVTrack achieves state-of-the-art results while operating at 200 FPS, enabling real-time applicability.
  arXiv  Detail & Related papers  (2023-09-05T12:42:26Z)
• Large Scale Real-World Multi-Person Tracking [68.27438015329807]
  This paper presents a new large scale multi-person tracking dataset -- textttPersonPath22. It is over an order of magnitude larger than currently available high quality multi-object tracking datasets such as MOT17, HiEve, and MOT20.
  arXiv  Detail & Related papers  (2022-11-03T23:03:13Z)
• Detection, Tracking, and Counting Meets Drones in Crowds: A Benchmark [97.07865343576361]
  We construct a benchmark with a new drone-captured largescale dataset, named as DroneCrowd. We annotate 20,800 people trajectories with 4.8 million heads and several video-level attributes. We design the Space-Time Neighbor-Aware Network (STNNet) as a strong baseline to solve object detection, tracking and counting jointly in dense crowds.
  arXiv  Detail & Related papers  (2021-05-06T04:46:14Z)
• Learnable Online Graph Representations for 3D Multi-Object Tracking [156.58876381318402]
  We propose a unified and learning based approach to the 3D MOT problem. We employ a Neural Message Passing network for data association that is fully trainable. We show the merit of the proposed approach on the publicly available nuScenes dataset by achieving state-of-the-art performance of 65.6% AMOTA and 58% fewer ID-switches.
  arXiv  Detail & Related papers  (2021-04-23T17:59:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.