FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking

• URL: http://arxiv.org/abs/2606.29783v1
• Date: Mon, 29 Jun 2026 05:03:57 GMT
• Title: FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking
• Abstract summary: FalconTrack is a unified perception-and-tracking framework for vision-based aerial tracking.<n>It produces 10k labeled images in under 20 minutes and 96-100% class accuracy in zero-shot sim-to-real transfer.<n>The onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments.
• Score: 5.430107038361476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack, a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer. FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this dataset, we train a multi-head perception module with staged learning and reprojection consistency, and fuse its outputs with class-conditioned dynamics priors in an EKF for tracking. Our perception model outperforms two baselines and reaches 96-100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios.

Related papers

• Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator [70.34667754804168]
  Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world.<n>We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences.
  arXiv  Detail & Related papers  (2026-07-07T02:42:41Z)
• GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning [55.272458304179025]
  GS-Playground is a multi-modal simulation framework designed to accelerate end-to-end perceptual learning.<n>Our system achieves a breakthrough throughput of 104 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL.
  arXiv  Detail & Related papers  (2026-04-28T10:05:39Z)
• Learning to Track Any Points from Human Motion [55.831218129679144]
  We propose an automated pipeline to generate pseudo-labeled training data for point tracking.<n>A point tracking model trained on AnthroTAP achieves annotated state-of-the-art performance on the TAP-Vid benchmark.
  arXiv  Detail & Related papers  (2025-07-08T17:59:58Z)
• FalconWing: An Ultra-Light Indoor Fixed-Wing UAV Platform for Vision-Based Autonomy [3.7383337849330336]
  FalconWing is an ultra-light (150 g) indoor fixed-wing UAV platform for vision-based autonomy.<n>FalconWing couples a lightweight hardware stack (137g airframe with a 9g camera) and offboard with a software stack featuring a 3D Gaussian Splat (GSplat) simulator for developing and evaluating vision-based controllers.<n>We will release hardware designs, GSplat scenes, and dynamics models upon publication to make FalconWing an open-source flight kit for engineering students and research labs.
  arXiv  Detail & Related papers  (2025-05-02T16:47:05Z)
• HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving [48.84595398410479]
  HUGSIM is a closed-loop, photo-realistic, and real-time simulator for evaluating autonomous driving algorithms.<n>We tackle challenges of novel view synthesis in closed-loop scenarios, including viewpoint extrapolation and 360-degree vehicle rendering.<n> HUGSIM offers a comprehensive benchmark across more than 70 sequences from KITTI-360, nuScenes, and PandaSet, along with over 400 varying scenarios.
  arXiv  Detail & Related papers  (2024-12-02T17:07:59Z)
• Open-World Drone Active Tracking with Goal-Centered Rewards [62.21394499788672]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations.<n>We propose DAT, the first open-world drone active air-to-ground tracking benchmark.<n>We also propose GC-VAT, which aims to improve the performance of drone tracking targets in complex scenarios.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• An Effective Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds [50.19288542498838]
  3D single object tracking in LiDAR point clouds (LiDAR SOT) plays a crucial role in autonomous driving. Current approaches all follow the Siamese paradigm based on appearance matching. We introduce a motion-centric paradigm to handle LiDAR SOT from a new perspective.
  arXiv  Detail & Related papers  (2023-03-21T17:28:44Z)
• RARA: Zero-shot Sim2Real Visual Navigation with Following Foreground Cues [42.998649025215045]
  We tackle the specific case of camera-based navigation, formulating it as following a visual cue in the foreground with arbitrary backgrounds. The goal is to train a visual agent on data captured in an empty simulated environment except for this foreground cue and test this model directly in a visually diverse real world.
  arXiv  Detail & Related papers  (2022-01-08T09:53:21Z)
• DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
  We introduce a novel high-quality neural simulator referred to as DriveGAN. DriveGAN achieves controllability by disentangling different components without supervision. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
  arXiv  Detail & Related papers  (2021-04-30T15:30:05Z)
• Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
  A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
  arXiv  Detail & Related papers  (2021-03-12T15:30:02Z)
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving [27.948417322786575]
  We present a simple yet effective approach to generate realistic scenario sensor data. Our approach uses texture-mapped surfels to efficiently reconstruct the scene from an initial vehicle pass or set of passes. We then leverage a SurfelGAN network to reconstruct realistic camera images for novel positions and orientations of the self-driving vehicle.
  arXiv  Detail & Related papers  (2020-05-08T04:01:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.