Flying in Clutter on Monocular RGB by Learning in 3D Radiance Fields with Domain Adaptation

• URL: http://arxiv.org/abs/2512.17349v1
• Date: Fri, 19 Dec 2025 08:44:06 GMT
• Title: Flying in Clutter on Monocular RGB by Learning in 3D Radiance Fields with Domain Adaptation
• Abstract summary: We propose a framework that couples the photorealism of 3D Gaussian Splatting (3DGS) environments with Adversarial Domain Adaptation.<n>By training in high-fidelity simulation while explicitly minimizing feature discrepancy, our method ensures the policy relies on domain-invariant cues.<n> Experimental results demonstrate that our policy achieves robust zero-shot transfer to the physical world, enabling safe and agile flight.
• Score: 9.881745654821922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern autonomous navigation systems predominantly rely on lidar and depth cameras. However, a fundamental question remains: Can flying robots navigate in clutter using solely monocular RGB images? Given the prohibitive costs of real-world data collection, learning policies in simulation offers a promising path. Yet, deploying such policies directly in the physical world is hindered by the significant sim-to-real perception gap. Thus, we propose a framework that couples the photorealism of 3D Gaussian Splatting (3DGS) environments with Adversarial Domain Adaptation. By training in high-fidelity simulation while explicitly minimizing feature discrepancy, our method ensures the policy relies on domain-invariant cues. Experimental results demonstrate that our policy achieves robust zero-shot transfer to the physical world, enabling safe and agile flight in unstructured environments with varying illumination.

Related papers

• Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator [70.34667754804168]
  Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world.<n>We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences.
  arXiv  Detail & Related papers  (2026-07-07T02:42:41Z)
• Trans2Occ: Voxel Occupancy Estimation and Grasp for Transparent Objects from Simulation to Reality [125.76063573012853]
  We present a framework for transparent object perception and manipulation based on single-view RGB input.<n>Our approach predicts voxel-space occupancy directly from a single image, providing a geometry-aware representation that supports downstream robotic grasping.
  arXiv  Detail & Related papers  (2026-06-01T06:59:13Z)
• GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields [10.10713417247828]
  GaussFly is a novel framework that explicitly decouples representation learning from policy optimization.<n>It enables robust and zero-shot policy transfer to unseen real-world environments with complex textures.
  arXiv  Detail & Related papers  (2026-04-06T18:10:52Z)
• Zero-Shot UAV Navigation in Forests via Relightable 3D Gaussian Splatting [11.31291385822484]
  UAV navigation in unstructured outdoor environments using passive monocular vision is hindered by the substantial visual domain gap between simulation and reality.<n>We propose a novel end-to-end reinforcement learning framework designed for effective zero-shot transfer to unstructured outdoors.<n>We show that a lightweight quadrotor achieves robust, collision-free navigation in complex forest environments at speeds up to 10 m/s.
  arXiv  Detail & Related papers  (2026-02-06T15:51:03Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion [25.440573256776133]
  This paper presents a Real-to-Sim-to-Real framework that generates and physically interactive "digital twin" simulation environments for visual navigation and locomotion learning.
  arXiv  Detail & Related papers  (2025-02-03T17:15:05Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles [131.2240621036954]
  We present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras. We demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle.
  arXiv  Detail & Related papers  (2021-11-23T18:58:10Z)
• Learning a State Representation and Navigation in Cluttered and Dynamic Environments [6.909283975004628]
  We present a learning-based pipeline to realise local navigation with a quadrupedal robot in cluttered environments. The robot is able to safely locomote to a target location based on frames from a depth camera without any explicit mapping of the environment. We show that our system can handle noisy depth images, avoid dynamic obstacles unseen during training, and is endowed with local spatial awareness.
  arXiv  Detail & Related papers  (2021-03-07T13:19:06Z)
• Transferable Active Grasping and Real Embodied Dataset [48.887567134129306]
  We show how to search for feasible viewpoints for grasping by the use of hand-mounted RGB-D cameras. A practical 3-stage transferable active grasping pipeline is developed, that is adaptive to unseen clutter scenes. In our pipeline, we propose a novel mask-guided reward to overcome the sparse reward issue in grasping and ensure category-irrelevant behavior.
  arXiv  Detail & Related papers  (2020-04-28T08:15:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.