UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models

• URL: http://arxiv.org/abs/2604.02241v1
• Date: Thu, 02 Apr 2026 16:33:38 GMT
• Title: UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models
• Abstract summary: Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks.<n>To benchmark multimodal tracking in such environments, we construct a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects.<n>We propose an improved VLA tracking model, UAV-Track VLA, built upon the $_0.5$ architecture.
• Score: 14.433739259355406
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks. In dynamic urban scenarios with complex semantic requirements, Vision-Language-Action (VLA) models show great promise due to their cross-modal fusion and continuous action generation capabilities. To benchmark multimodal tracking in such environments, we construct a dedicated evaluation benchmark and a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects. Furthermore, to address temporal feature redundancy and the lack of spatial geometric priors in existing VLA models, we propose an improved VLA tracking model, UAV-Track VLA. Built upon the $π_{0.5}$ architecture, our model introduces a temporal compression net to efficiently capture inter-frame dynamics. Additionally, a parallel dual-branch decoder comprising a spatial-aware auxiliary grounding head and a flow matching action expert is designed to decouple cross-modal features and generate fine-grained continuous actions. Systematic experiments in the CARLA simulator validate the superior end-to-end performance of our method. Notably, in challenging long-distance pedestrian tracking tasks, UAV-Track VLA achieves a 61.76\% success rate and 269.65 average tracking frames, significantly outperforming existing baselines. Furthermore, it demonstrates robust zero-shot generalization in unseen environments and reduces single-step inference latency by 33.4\% (to 0.0571s) compared to the original $π_{0.5}$, enabling highly efficient, real-time UAV control. Data samples and demonstration videos are available at: https://github.com/Hub-Tian/UAV-Track\_VLA.

Related papers

• CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization [5.936588226093352]
  We introduce CosFlyTrack, a large-scale multi-modal dataset for UAV visual tracking in urban environments.<n>The dataset provides approximately 12,000 expert and perturbed UAV trajectories generated from 6,000 pedestrian paths.<n>CosFlyTrack improves tracking performance to 78.3 to 95.6% SR@1 meter, a 53 to 69 percentage point gain over zero-shot baselines.
  arXiv  Detail & Related papers  (2026-05-18T02:49:58Z)
• A Topology-Aware Spatiotemporal Handover Framework for Continuous Multi-UAV Tracking [6.2002105625228845]
  This paper presents a real-time Multi-Camera Multi-Vehicle Tracking (MCMT) system designed to handle global identity persistence.<n>We introduce a lightweight Topology-Based Spatiotemporal Handover mechanism.<n> Experimental results on complex urban environments, including intersections and merging traffic, demonstrate a Handover Success Rate (HOSR) of 99.8% in continuous traffic flows.
  arXiv  Detail & Related papers  (2026-05-15T09:38:09Z)
• Architecture and evaluation protocol for transformer-based visual object tracking in UAV applications [0.0]
  Existing visual trackers either lack robustness in complex scenarios or are too computationally demanding for real-time embedded use.<n>We propose an Modular Asynchronous Tracking Architecture (MATA) that combines a transformer-based tracker with an Extended Kalman Filter.<n>We introduce a hardware-independent, embedded oriented evaluation protocol and a new metric called Normalized time to Failure (NT2F) to quantify how long a tracker can sustain a tracking sequence without external help.
  arXiv  Detail & Related papers  (2026-03-04T10:12:12Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos [58.156141601478794]
  Multi-object tracking (UAVT) aims to track multiple objects while maintaining consistent identities across frames of a given video.<n>Existing methods typically model motion cues and appearance separately, overlooking their interplay and resulting in suboptimal tracking performance.<n>We propose AMOT, which exploits appearance and motion cues through two key components: an Appearance-Motion Consistency (AMC) matrix and a Motion-aware Track Continuation (MTC) module.
  arXiv  Detail & Related papers  (2025-08-03T12:06:47Z)
• TrackVLA: Embodied Visual Tracking in the Wild [34.03604806748204]
  Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision.<n>Existing approaches typically address this challenge through a modular separation of recognition and planning.<n>We propose TrackVLA, a Vision-Language-Action model that learns the synergy between object recognition and trajectory planning.
  arXiv  Detail & Related papers  (2025-05-29T07:28:09Z)
• COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking [52.62149024881728]
  We propose a contrastive one-stage transformer fusion framework for vision-language (VL) tracking.<n>We introduce a contrastive alignment strategy that maximizes mutual information between a video and its corresponding language description.<n>By leveraging a visual-linguistic transformer, we establish an efficient multi-modal fusion and reasoning mechanism.
  arXiv  Detail & Related papers  (2025-04-02T03:12:38Z)
• MUST: The First Dataset and Unified Framework for Multispectral UAV Single Object Tracking [17.96400810834486]
  We introduce the first large-scale Multispectral UAV Single Object Tracking dataset (MUST)<n>MUST includes 250 video sequences spanning diverse environments and challenges.<n>We also propose a novel tracking framework, UNTrack, which encodes unified spectral, spatial, and temporal features from spectrum prompts.
  arXiv  Detail & Related papers  (2025-03-22T08:47:28Z)
• TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies [95.30717188630432]
  We introduce visual trace prompting to facilitate VLA models' spatial-temporal awareness for action prediction.<n>We develop a new TraceVLA model by finetuning OpenVLA on our own collected dataset of 150K robot manipulation trajectories.<n>We present a compact VLA model based on 4B Phi-3-Vision, pretrained on the Open-X-Embodiment and finetuned on our dataset.
  arXiv  Detail & Related papers  (2024-12-13T18:40:51Z)
• Learning Motion Blur Robust Vision Transformers for Real-Time UAV Tracking [14.382072224997074]
  Unmanned aerial vehicle (UAV) tracking is critical for applications like surveillance, search-and-rescue, and autonomous navigation.<n>The high-speed movement of UAVs and targets introduces unique challenges, including real-time processing demands and severe motion blur.<n>We propose an adaptive computation framework that dynamically exits Transformer blocks for real-time UAV tracking.
  arXiv  Detail & Related papers  (2024-07-07T14:10:04Z)
• Exploring Dynamic Transformer for Efficient Object Tracking [58.120191254379854]
  We propose DyTrack, a dynamic transformer framework for efficient tracking.<n>DyTrack automatically learns to configure proper reasoning routes for various inputs, gaining better utilization of the available computational budget.<n>Experiments on multiple benchmarks demonstrate that DyTrack achieves promising speed-precision trade-offs with only a single model.
  arXiv  Detail & Related papers  (2024-03-26T12:31:58Z)
• Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching [67.02962970820505]
  We introduce "tracking-by-detection" into Video Object (VOS) We propose a new temporal aggregation network and a novel dynamic time-evolving template matching mechanism to achieve significantly improved performance. We achieve new state-of-the-art performance on the DAVIS benchmark without complicated bells and whistles in both speed and accuracy, with a speed of 0.14 second per frame and J&F measure of 75.9% respectively.
  arXiv  Detail & Related papers  (2020-07-11T05:44:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.