Digital Twin Supervised Reinforcement Learning Framework for Autonomous Underwater Navigation

• URL: http://arxiv.org/abs/2512.10925v1
• Date: Thu, 11 Dec 2025 18:52:42 GMT
• Title: Digital Twin Supervised Reinforcement Learning Framework for Autonomous Underwater Navigation
• Authors: Zamirddine Mari, Mohamad Motasem Nawaf, Pierre Drap,
• Abstract summary: This article investigates issues through the case of the BlueROV2, an open platform widely used for scientific experimentation.<n>We propose a deep reinforcement learning approach based on the Proximal Policy Optimization (PPO) algorithm.<n>Results show that the PPO policy consistently outperforms DWA in highly cluttered environments.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous navigation in underwater environments remains a major challenge due to the absence of GPS, degraded visibility, and the presence of submerged obstacles. This article investigates these issues through the case of the BlueROV2, an open platform widely used for scientific experimentation. We propose a deep reinforcement learning approach based on the Proximal Policy Optimization (PPO) algorithm, using an observation space that combines target-oriented navigation information, a virtual occupancy grid, and ray-casting along the boundaries of the operational area. The learned policy is compared against a reference deterministic kinematic planner, the Dynamic Window Approach (DWA), commonly employed as a robust baseline for obstacle avoidance. The evaluation is conducted in a realistic simulation environment and complemented by validation on a physical BlueROV2 supervised by a 3D digital twin of the test site, helping to reduce risks associated with real-world experimentation. The results show that the PPO policy consistently outperforms DWA in highly cluttered environments, notably thanks to better local adaptation and reduced collisions. Finally, the experiments demonstrate the transferability of the learned behavior from simulation to the real world, confirming the relevance of deep RL for autonomous navigation in underwater robotics.

Related papers

• IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation [56.43007596544299]
  IndustryNav is the first dynamic industrial navigation benchmark for active spatial reasoning.<n>A study of nine state-of-the-art Visual Large Language Models reveals that closed-source models maintain a consistent advantage.
  arXiv  Detail & Related papers  (2025-11-21T16:48:49Z)
• DUViN: Diffusion-Based Underwater Visual Navigation via Knowledge-Transferred Depth Features [47.88998580611257]
  We propose a Diffusion-based Underwater Visual Navigation policy via knowledge-transferred depth features, named DUViN.<n>DuViN guides the vehicle to avoid obstacles and maintain a safe and perception awareness altitude relative to the terrain without relying on pre-built maps.<n> Experiments in both simulated and real-world underwater environments demonstrate the effectiveness and generalization of our approach.
  arXiv  Detail & Related papers  (2025-09-03T03:43:12Z)
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning [59.88543114325153]
  We introduce the Seeing-to-Experiencing framework to scale the capability of navigation foundation models with reinforcement learning.<n>S2E combines the strengths of pre-training on videos and post-training through RL.<n>We establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3DGS reconstructions of real-world scenes.
  arXiv  Detail & Related papers  (2025-07-29T17:26:10Z)
• Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigation [55.02966123945644]
  We propose a hierarchical control framework leveraging neural network verification techniques to design control barrier functions (CBFs) and policy correction mechanisms.<n>Our approach relies on probabilistic enumeration to identify unsafe regions of operation, which are then used to construct a safe CBF-based control layer.<n>These experiments demonstrate the ability of the proposed solution to correct unsafe actions while preserving efficient navigation behavior.
  arXiv  Detail & Related papers  (2025-04-30T13:47:25Z)
• Depth-Constrained ASV Navigation with Deep RL and Limited Sensing [43.785833390490446]
  We propose a reinforcement learning framework for ASV navigation under depth constraints.<n>To enhance environmental awareness, we integrate GP regression into the RL framework.<n>We demonstrate effective sim-to-real transfer, ensuring that trained policies generalize well to real-world aquatic conditions.
  arXiv  Detail & Related papers  (2025-04-25T10:56:56Z)
• Hierarchical Reinforcement Learning for Safe Mapless Navigation with Congestion Estimation [7.339743259039457]
  This paper introduces a safe mapless navigation framework utilizing hierarchical reinforcement learning (HRL) to enhance navigation through such areas.<n>The findings demonstrate that our HRL-based navigation framework excels in both static and dynamic scenarios.<n>We implement the HRL-based navigation framework on a TurtleBot3 robot for physical validation experiments.
  arXiv  Detail & Related papers  (2025-03-15T08:03:50Z)
• RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation [9.25068777307471]
  This paper introduces a learning-based visual planner for agile drone flight in cluttered environments.<n>The proposed planner generates collision-free waypoints in milliseconds, enabling drones to perform agile maneuvers in complex environments without building separate perception, mapping, and planning modules.
  arXiv  Detail & Related papers  (2025-02-04T06:42:08Z)
• Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges [0.46873264197900916]
  The predominant use of sonar in underwater environments, characterized by limited training data and inherent noise, poses challenges to model robustness.<n>This paper studies sonar-based perception task models, such as classification, object detection, segmentation, and SLAM.<n>It systematizes sonar-based state-of-the-art datasets, simulators, and robustness methods such as neural network verification, out-of-distribution, and adversarial attacks.
  arXiv  Detail & Related papers  (2024-12-16T15:03:08Z)
• Vision-Based Deep Reinforcement Learning of UAV Autonomous Navigation Using Privileged Information [6.371251946803415]
  DPRL is an end-to-end policy designed to address the challenge of high-speed autonomous UAV navigation under partially observable environmental conditions.<n>We leverage an asymmetric Actor-Critic architecture to provide the agent with privileged information during training.<n>We conduct extensive simulations across various scenarios, benchmarking our DPRL algorithm against the state-of-the-art navigation algorithms.
  arXiv  Detail & Related papers  (2024-12-09T09:05:52Z)
• Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning [53.3760591018817]
  We propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and Deep Reinforcement Learning. Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques. Our empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results.
  arXiv  Detail & Related papers  (2024-05-30T23:20:23Z)
• Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation [78.17108227614928]
  We propose a benchmark environment for Safe Reinforcement Learning focusing on aquatic navigation. We consider a value-based and policy-gradient Deep Reinforcement Learning (DRL) We also propose a verification strategy that checks the behavior of the trained models over a set of desired properties.
  arXiv  Detail & Related papers  (2021-12-16T16:53:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.