Mastering Diverse, Unknown, and Cluttered Tracks for Robust Vision-Based Drone Racing

• URL: http://arxiv.org/abs/2512.09571v2
• Date: Thu, 11 Dec 2025 04:26:24 GMT
• Title: Mastering Diverse, Unknown, and Cluttered Tracks for Robust Vision-Based Drone Racing
• Abstract summary: Most reinforcement learning methods for drone racing target fixed, obstacle-free tracks, leaving the generalization to unknown, cluttered environments largely unaddressed.<n>This challenge stems from the need to balance racing speed and collision avoidance, limited feasible space causing policy exploration trapped in local optima during training.<n>We propose a two-phase learning framework: an initial soft-collision training phase that preserves policy exploration for high-speed flight, followed by a hard-collision refinement phase that enforces robust obstacle avoidance.
• Score: 22.63301233637327
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Most reinforcement learning(RL)-based methods for drone racing target fixed, obstacle-free tracks, leaving the generalization to unknown, cluttered environments largely unaddressed. This challenge stems from the need to balance racing speed and collision avoidance, limited feasible space causing policy exploration trapped in local optima during training, and perceptual ambiguity between gates and obstacles in depth maps-especially when gate positions are only coarsely specified. To overcome these issues, we propose a two-phase learning framework: an initial soft-collision training phase that preserves policy exploration for high-speed flight, followed by a hard-collision refinement phase that enforces robust obstacle avoidance. An adaptive, noise-augmented curriculum with an asymmetric actor-critic architecture gradually shifts the policy's reliance from privileged gate-state information to depth-based visual input. We further impose Lipschitz constraints and integrate a track-primitive generator to enhance motion stability and cross-environment generalization. We evaluate our framework through extensive simulation and ablation studies, and validate it in real-world experiments on a computationally constrained quadrotor. The system achieves agile flight while remaining robust to gate-position errors, developing a generalizable drone racing framework with the capability to operate in diverse, partially unknown and cluttered environments. https://yufengsjtu.github.io/MasterRacing.github.io/

Related papers

• Local Path Planning and Obstacle Avoidance for an Omnicopter Platform [5.329751638083761]
  This paper presents a real-time local planning and obstacle avoidance module for an omnidirectional multirotor (omnicopter)<n>Our method achieves real-time feasibility through (i) local-map voxelisation, (ii) a compact sphere-based approximation of the vehicle geometry, and (iii) adaptive velocity sampling in the 6D search space.<n>We evaluate our approach in simulation across computational stress tests, dense-waypoint path tracking, and static/unknown obstacle scenarios.
  arXiv  Detail & Related papers  (2026-09-03T10:25:14Z)
• Bridging Performance and Generalization in Reinforcement Learning for Agile Flight [26.465258512733126]
  We propose a framework for zero-shot generalization in agile flight for RL-based drone racing.<n>Our method achieves strong zero-shot performance across a wide range of unseen racetracks in the real world.
  arXiv  Detail & Related papers  (2026-06-25T17:54:36Z)
• Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking [78.30338456206984]
  We introduce a novel framework for training general-purpose 3D foothold-tracking policies.<n>By dynamically providing footstep support through a goal sampler, this method enables the learned policy to be agnostic to specific terrains.<n>Our new target representation effectively mitigates challenges arising in the real world, such as noisy and inaccurate pose estimation and foot contact estimation.
  arXiv  Detail & Related papers  (2026-06-06T16:44:56Z)
• Learning Reactive Dexterous Grasping via Hierarchical Task-Space RL Planning and Joint-Space QP Control [50.28263951510334]
  We propose a hybrid hierarchical control framework for reactive dexterous grasping.<n>The proposed framework explicitly decouples high-level spatial intent from low-level joint execution.<n>We extensively validate the proposed framework through a rigorous simulation-to-reality pipeline.
  arXiv  Detail & Related papers  (2026-05-05T04:49:38Z)
• Vector Field Augmented Differentiable Policy Learning for Vision-Based Drone Racing [8.010590009724636]
  Differentiable-physics-based policy learning has recently demonstrated high sample efficiency and remarkable performance across various tasks.<n>We propose DiffRacing, a novel vector field-augmented differentiable policy learning framework.<n>We show that DiffRacing achieves superior sample efficiency, faster convergence, and robust flight performance.
  arXiv  Detail & Related papers  (2026-03-09T06:49:41Z)
• Vision-Guided MPPI for Agile Drone Racing: Navigating Arbitrary Gate Poses via Neural Signed Distance Fields [8.66703842496129]
  Drone racing requires tight coupling of perception, planning, and control under extreme agility.<n>Recent approaches rely on precomputed spatial reference trajectories or explicit 6-DoF gate pose estimation.<n>We propose a vision guided optimal control framework that enables reference-free agile flight through arbitrarily placed and oriented gates.
  arXiv  Detail & Related papers  (2026-03-07T13:18:07Z)
• Curriculum Reinforcement Learning for Quadrotor Racing with Random Obstacles [12.806994271928483]
  We propose a novel vision-based curriculum reinforcement learning framework for training a robust controller capable of addressing unseen obstacles in drone racing.<n>Our end-to-end control policy is implemented as a single network, allowing high-speed flight of quadrotors in environments with variable obstacles.
  arXiv  Detail & Related papers  (2026-02-27T13:56:59Z)
• NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environments [56.35569661650558]
  We introduce NOVA, a fully onboard, object-centric framework that enables robust target tracking and collision-aware navigation.<n>Rather than constructing a global map, NOVA formulates perception, estimation, and control entirely in the target's reference frame.<n>We validate NOVA across challenging real-world scenarios, including urban mazes, forest trails, and repeated transitions through buildings with intermittent GPS loss.
  arXiv  Detail & Related papers  (2025-06-23T14:28:30Z)
• Open-World Drone Active Tracking with Goal-Centered Rewards [62.21394499788672]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations.<n>We propose DAT, the first open-world drone active air-to-ground tracking benchmark.<n>We also propose GC-VAT, which aims to improve the performance of drone tracking targets in complex scenarios.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• Learning Generalizable Policy for Obstacle-Aware Autonomous Drone Racing [0.0]
  This study addresses the challenge of developing a generalizable obstacle-aware drone racing policy. We propose applying domain randomization on racing tracks and obstacle configurations before every rollout. The proposed randomization strategy is shown to be effective through simulated experiments where drones reach speeds of up to 70 km/h.
  arXiv  Detail & Related papers  (2024-11-06T20:25:43Z)
• RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes [57.319845580050924]
  We propose a reinforcement learning framework that combines risk-sensitive control with an adaptive action space curriculum. We show that our algorithm is capable of learning high-speed policies for a real-world off-road driving task.
  arXiv  Detail & Related papers  (2024-05-07T23:32:36Z)
• Learning High-Speed Flight in the Wild [101.33104268902208]
  We propose an end-to-end approach that can autonomously fly quadrotors through complex natural and man-made environments at high speeds. The key principle is to directly map noisy sensory observations to collision-free trajectories in a receding-horizon fashion. By simulating realistic sensor noise, our approach achieves zero-shot transfer from simulation to challenging real-world environments.
  arXiv  Detail & Related papers  (2021-10-11T09:43:11Z)
• Vision-Based Mobile Robotics Obstacle Avoidance With Deep Reinforcement Learning [49.04274612323564]
  Obstacle avoidance is a fundamental and challenging problem for autonomous navigation of mobile robots. In this paper, we consider the problem of obstacle avoidance in simple 3D environments where the robot has to solely rely on a single monocular camera. We tackle the obstacle avoidance problem as a data-driven end-to-end deep learning approach.
  arXiv  Detail & Related papers  (2021-03-08T13:05:46Z)
• Autonomous Off-road Navigation over Extreme Terrains with Perceptually-challenging Conditions [7.514178230130502]
  We propose a framework for resilient autonomous computation in perceptually challenging environments with mobility-stressing elements. We propose a fast settling algorithm to generate robust multi-fidelity traversability estimates in real-time. The proposed approach was deployed on multiple physical systems including skid-steer and tracked robots, a high-speed RC car and legged robots.
  arXiv  Detail & Related papers  (2021-01-26T22:13:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.