Sim-to-Real Transfer and Robustness Evaluation of Reinforcement Learning Control with Integrated Perception on an ASV for Floating Waste Capture

• URL: http://arxiv.org/abs/2605.02529v1
• Date: Mon, 04 May 2026 12:29:15 GMT
• Title: Sim-to-Real Transfer and Robustness Evaluation of Reinforcement Learning Control with Integrated Perception on an ASV for Floating Waste Capture
• Abstract summary: We present a field-validated system that combines camera-based polarimetric perception with a lightweight DRL-based controller for floating-waste detection and capture.<n>Camera detections are converted into water-surface target points and tracked by a controller trained entirely in simulation and deployed directly on a retrofitted ASV platform.
• Score: 4.05330458826742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous surface vessels for floating-waste removal operate under varying hydrodynamics, external disturbances, and challenging water-surface perception. We present a field-validated system that combines camera-based polarimetric perception with a lightweight DRL-based controller for floating-waste detection and capture. Camera detections are converted into water-surface target points and tracked by a controller trained entirely in simulation and deployed directly on a retrofitted ASV platform. Our main contribution is a sim-to-real testing methodology that combines a two-stage simulation protocol with a perception abstraction module designed to mimic real camera behavior, enabling reproducible field trials and explicit evaluation of the sim-to-real gap. We apply this framework in matched simulation and field experiments across 14 disturbance regimes to expose failure modes and evaluate robustness. The results show centimeter-level terminal accuracy and indicate robust control performance under the evaluated perturbation regimes. The main source of degradation is insufficient actuation-model fidelity. We also demonstrate the system in a search-and-capture application using real camera detections in real-world conditions over areas of up to $450~m^2$. The study distills practical lessons for reliable transfer, including improved actuation-model fidelity, targeted domain randomization, and careful management of latency and timestamps across modules, while highlighting remaining challenges.

Related papers

• Task-Relevant Feature-Dynamics Fidelity Enables Zero-Shot Sim-to-Real Transfer for Robotic Ultrasound Scanning [58.25007580703646]
  We develop a TR-oriented ultrasound simulator that combines a shared structural intermediate domain, trajectory-level fixed noise, and few-step conditional flow generation.<n>In phantom experiments, a policy trained exclusively in simulation succeeded in 390 of 400 zero-shot deployments across four target planes.
  arXiv  Detail & Related papers  (2026-08-30T02:37:00Z)
• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• Learning a System-Level Surrogate for Hydraulic Excavators: A Simulation-to-Real LSTM Approach [19.924870252624647]
  This paper proposes a simulation-to-real framework for learning a system-level digital surrogate.<n>Instead of modeling internal dynamics, the excavator is treated as an input-output operator.<n>The proposed model can serve as a drop-in surrogate for both simulation and physical systems.
  arXiv  Detail & Related papers  (2026-07-17T06:04:08Z)
• Uncertainty-Aware Adaptive Sensor Fusion for Autonomous Navigation [0.0]
  This work introduces a hybrid deep learning approach integrated with an Unscented Kalman Filter (UKF) to enhance pose estimation accuracy in Visual-Inertial Odometry (VIO) for autonomous navigation.<n>The proposed model employs a Vision Transformer (ViT) network to effectively capture temporal dependencies from inertial measurement unit (IMU) data.<n>A novel uncertainty-aware loss function is proposed to explicitly incorporate prediction uncertainty into the learning process, enabling robust and accurate navigation under noisy, incomplete, or unreliable sensor inputs.
  arXiv  Detail & Related papers  (2026-06-03T20:59:42Z)
• MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation [65.068801413044]
  Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams.<n>We introduce MoASE, a mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture.
  arXiv  Detail & Related papers  (2026-05-18T01:52:12Z)
• Instantaneous Planning, Control and Safety for Navigation in Unknown Underwater Spaces [0.21108097398435333]
  Navigating autonomous underwater vehicles (AUVs) in unknown environments is significantly challenging.<n>These factors pose challenges in accurate global localization, reliable communication, and obstacle avoidance.<n>We propose an integrated planning and control framework that leverages real time sensor data to dynamically induce closed loop AUV trajectories.
  arXiv  Detail & Related papers  (2026-04-07T01:32:29Z)
• Compressive sensing inspired self-supervised single-pixel imaging [48.79613794195208]
  SISTA-Net is a compressive sensing-inspired self-supervised method for single-pixel imaging.<n>We leverage deep nonlinear networks as adaptive sparse transforms combined with a learnable soft-thresholding operator to impose explicit physical sparsity in the latent domain.<n>Experiments on multiple simulation scenarios demonstrate that SISTA-Net outperforms state-of-the-art methods by 2.6 dB in PSNR.
  arXiv  Detail & Related papers  (2026-03-31T13:31:43Z)
• ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation [72.78362530982109]
  ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, is a framework that decouples exploration from commitment.<n>We show that naive LLM-based simulators struggle to capture rare but high-impact failure modes.<n>We introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions.
  arXiv  Detail & Related papers  (2026-02-02T06:33:22Z)
• Towards reliable subsea object recovery: a simulation study of an auv with a suction-actuated end effector [1.1984905847118061]
  This paper presents a simulation-based study of a complete autonomous subsea object recovery mission using a Hadal Small Vehicle (HSV)<n>The Stonefish simulator is used to model realistic vehicle dynamics, hydrodynamic disturbances, sensing, and interaction with a target object under hadal-like conditions.<n>The HSV autonomously descends from the sea surface to 6,000 m, performs structured seafloor coverage, detects a target object, and executes a suction-based recovery.
  arXiv  Detail & Related papers  (2026-01-03T07:55:23Z)
• A2VISR: An Active and Adaptive Ground-Aerial Localization System Using Visual Inertial and Single-Range Fusion [17.59898575353575]
  It's a practical approach using the ground-aerial collaborative system to enhance the localization robustness of flying robots in cluttered environments.<n>We improve the ground-aerial localization framework in a more comprehensive manner, which integrates active vision, single-ranging, inertialometry, and optical flow.<n>The proposed approach achieves robust online localization, with an average root mean square error of approximately 0.09 m, while maintaining resilience to capture loss and sensor failures.
  arXiv  Detail & Related papers  (2025-12-18T10:07:06Z)
• A Sensor-Aware Phenomenological Framework for Lidar Degradation Simulation and SLAM Robustness Evaluation [1.9228699691035367]
  Lidar-based SLAM systems are highly sensitive to adverse conditions such as noise, and field-of-view (FoV) degradation.<n>This paper presents a sensor-aware framework for simulating interpretable lidar degradations directly on real point clouds.
  arXiv  Detail & Related papers  (2025-12-09T14:41:20Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)
• Testing Rare Downstream Safety Violations via Upstream Adaptive Sampling of Perception Error Models [20.815131169609316]
  This paper combines perception error models -- surrogates for a sensor-based detection system -- with state-dependent adaptive importance sampling. Experiments with an autonomous braking system equipped with an RGB obstacle-detector show that our method can calculate accurate failure probabilities.
  arXiv  Detail & Related papers  (2022-09-20T12:26:06Z)
• Visual-tactile sensing for Real-time liquid Volume Estimation in Grasping [58.50342759993186]
  We propose a visuo-tactile model for realtime estimation of the liquid inside a deformable container. We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor. The robotic system is well controlled and adjusted based on the estimation model in real time.
  arXiv  Detail & Related papers  (2022-02-23T13:38:31Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• Transferable Active Grasping and Real Embodied Dataset [48.887567134129306]
  We show how to search for feasible viewpoints for grasping by the use of hand-mounted RGB-D cameras. A practical 3-stage transferable active grasping pipeline is developed, that is adaptive to unseen clutter scenes. In our pipeline, we propose a novel mask-guided reward to overcome the sparse reward issue in grasping and ensure category-irrelevant behavior.
  arXiv  Detail & Related papers  (2020-04-28T08:15:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.