Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics

• URL: http://arxiv.org/abs/2503.10949v1
• Date: Thu, 13 Mar 2025 23:28:11 GMT
• Title: Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics
• Authors: Josip Josifovski, Shangding Gu, Mohammadhossein Malmir, Haoliang Huang, Sayantan Auddy, Nicolás Navarro-Guerrero, Costas Spanos, Alois Knoll,
• Abstract summary: Domain randomization is a technique to facilitate the transfer of policies from simulation to real-world robotic applications.<n>We propose a method to enable safe deployment-time policy adaptation in real-world robot control.
• Score: 3.7491742648742568
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Domain randomization has emerged as a fundamental technique in reinforcement learning (RL) to facilitate the transfer of policies from simulation to real-world robotic applications. Many existing domain randomization approaches have been proposed to improve robustness and sim2real transfer. These approaches rely on wide randomization ranges to compensate for the unknown actual system parameters, leading to robust but inefficient real-world policies. In addition, the policies pretrained in the domain-randomized simulation are fixed after deployment due to the inherent instability of the optimization processes based on RL and the necessity of sampling exploitative but potentially unsafe actions on the real system. This limits the adaptability of the deployed policy to the inevitably changing system parameters or environment dynamics over time. We leverage safe RL and continual learning under domain-randomized simulation to address these limitations and enable safe deployment-time policy adaptation in real-world robot control. The experiments show that our method enables the policy to adapt and fit to the current domain distribution and environment dynamics of the real system while minimizing safety risks and avoiding issues like catastrophic forgetting of the general policy found in randomized simulation during the pretraining phase. Videos and supplementary material are available at https://safe-cda.github.io/.

Related papers

• Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator [50.191655141020505]
  Reinforcement Learning (RL) has demonstrated impressive capabilities in robotic control but remains challenging due to high sample complexity, safety concerns, and the sim-to-real gap. We introduce Offline Robotic World Model (RWM-O), a model-based approach that explicitly estimates uncertainty to improve policy learning without reliance on a physics simulator.
  arXiv  Detail & Related papers  (2025-04-23T12:58:15Z)
• Post-Convergence Sim-to-Real Policy Transfer: A Principled Alternative to Cherry-Picking [5.027571997864706]
  This paper addresses the post-convergence sim-to-real transfer problem by introducing a worst-case performance transference optimization approach. Experiments demonstrate its effectiveness in transferring RL-based locomotion policies from simulation to real-world laboratory tests.
  arXiv  Detail & Related papers  (2025-04-21T19:48:05Z)
• Neural Fidelity Calibration for Informative Sim-to-Real Adaptation [10.117298045153564]
  Deep reinforcement learning can seamlessly transfer agile locomotion and navigation skills from the simulator to real world. However, bridging the sim-to-real gap with domain randomization or adversarial methods often demands expert physics knowledge to ensure policy robustness. We propose Neural Fidelity (NFC), a novel framework that employs conditional score-based diffusion models to calibrate simulator physical coefficients and residual fidelity domains online during robot execution.
  arXiv  Detail & Related papers  (2025-04-11T15:12:12Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization [10.789649934346004]
  We propose a sample-efficient method named cyclic policy distillation (CPD) CPD divides the range of randomized parameters into several small sub-domains and assigns a local policy to each one. All of the learned local policies are distilled into a global policy for sim-to-real transfers.
  arXiv  Detail & Related papers  (2022-07-29T09:22:53Z)
• Uncertainty Aware System Identification with Universal Policies [45.44896435487879]
  Sim2real transfer is concerned with transferring policies trained in simulation to potentially noisy real world environments. We propose Uncertainty-aware policy search (UncAPS), where we use Universal Policy Network (UPN) to store simulation-trained task-specific policies. We then employ robust Bayesian optimisation to craft robust policies for the given environment by combining relevant UPN policies in a DR like fashion.
  arXiv  Detail & Related papers  (2022-02-11T18:27:23Z)
• SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulation [12.778412161239466]
  Policy learned in the simulation may not always generate a safe behaviour on the real robot. In this work, we introduce a novel learning algorithm called SafeAPT that leverages a diverse repertoire of policies evolved in the simulation. We show that SafeAPT finds high-performance policies within a few minutes in the real world while minimizing safety violations during the interactions.
  arXiv  Detail & Related papers  (2022-01-27T16:40:36Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization [63.09932240840656]
  We show how to pre-train a perception encoder that already provides an embedding invariant to the randomization. We demonstrate this yields consistently improved results on a randomized version of DeepMind control suite tasks and a stacking environment on arbitrary backgrounds with zero-shot transfer to a physical robot.
  arXiv  Detail & Related papers  (2021-04-29T14:54:11Z)
• Improving Robustness via Risk Averse Distributional Reinforcement Learning [13.467017642143581]
  Robustness is critical when the policies are trained in simulations instead of real world environment. We propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation.
  arXiv  Detail & Related papers  (2020-05-01T20:03:10Z)
• Data-efficient Domain Randomization with Bayesian Optimization [34.854609756970305]
  When learning policies for robot control, the required real-world data is typically prohibitively expensive to acquire. BayRn is a black-box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution. Our results show that BayRn is able to perform sim-to-real transfer, while significantly reducing the required prior knowledge.
  arXiv  Detail & Related papers  (2020-03-05T07:48:31Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.