Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation

• URL: http://arxiv.org/abs/2507.06111v1
• Date: Tue, 08 Jul 2025 15:51:57 GMT
• Title: Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation
• Authors: Mohamad H. Danesh, Maxime Wabartha, Stanley Wu, Joelle Pineau, Hsiu-Chin Lin,
• Abstract summary: Uncertainty-Aware RL (UARL) is a novel framework that prioritizes safety during training by addressing Out-Of-Distribution (OOD) detection and policy adaptation without requiring direct interactions in target domain.<n>We evaluate UARL on MuJoCo benchmarks and a quadrupedal robot, demonstrating its effectiveness in reliable OOD detection, improved performance, and enhanced sample efficiency compared to baselines.
• Score: 26.241756408576684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deploying reinforcement learning (RL) policies in real-world involves significant challenges, including distribution shifts, safety concerns, and the impracticality of direct interactions during policy refinement. Existing methods, such as domain randomization (DR) and off-dynamics RL, enhance policy robustness by direct interaction with the target domain, an inherently unsafe practice. We propose Uncertainty-Aware RL (UARL), a novel framework that prioritizes safety during training by addressing Out-Of-Distribution (OOD) detection and policy adaptation without requiring direct interactions in target domain. UARL employs an ensemble of critics to quantify policy uncertainty and incorporates progressive environmental randomization to prepare the policy for diverse real-world conditions. By iteratively refining over high-uncertainty regions of the state space in simulated environments, UARL enhances robust generalization to the target domain without explicitly training on it. We evaluate UARL on MuJoCo benchmarks and a quadrupedal robot, demonstrating its effectiveness in reliable OOD detection, improved performance, and enhanced sample efficiency compared to baselines.

Related papers

• Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics [3.7491742648742568]
  Domain randomization is a technique to facilitate the transfer of policies from simulation to real-world robotic applications.<n>We propose a method to enable safe deployment-time policy adaptation in real-world robot control.
  arXiv  Detail & Related papers  (2025-03-13T23:28:11Z)
• Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling [84.00480999255628]
  Reinforcement Learning algorithms for safety alignment of Large Language Models (LLMs) encounter the challenge of distribution shift.<n>Current approaches typically address this issue through online sampling from the target policy.<n>We propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals.
  arXiv  Detail & Related papers  (2025-03-13T06:40:34Z)
• Mind the Gap: Towards Generalizable Autonomous Penetration Testing via Domain Randomization and Meta-Reinforcement Learning [15.619925926862235]
  GAP is a generalizable autonomous pentesting framework.<n>It aims to realizes efficient policy training in realistic environments.<n>It also trains agents capable of drawing inferences about other cases from one instance.
  arXiv  Detail & Related papers  (2024-12-05T11:24:27Z)
• Random Policy Enables In-Context Reinforcement Learning within Trust Horizons [2.52299400625445]
  State-Action Distillation (SAD) generates an effective pretraining dataset guided solely by random policies.<n>SAD outperforms the best baseline by 236.3% in the offline evaluation and by 135.2% in the online evaluation.
  arXiv  Detail & Related papers  (2024-10-25T21:46:25Z)
• Certifying Safety in Reinforcement Learning under Adversarial Perturbation Attacks [23.907977144668838]
  We propose a partially-supervised reinforcement learning (PSRL) framework that takes advantage of an additional assumption that the true state of the POMDP is known at training time. We present the first approach for certifying safety of PSRL policies under adversarial input perturbations, and two adversarial training approaches that make direct use of PSRL.
  arXiv  Detail & Related papers  (2022-12-28T22:33:38Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• Uncertainty Aware System Identification with Universal Policies [45.44896435487879]
  Sim2real transfer is concerned with transferring policies trained in simulation to potentially noisy real world environments. We propose Uncertainty-aware policy search (UncAPS), where we use Universal Policy Network (UPN) to store simulation-trained task-specific policies. We then employ robust Bayesian optimisation to craft robust policies for the given environment by combining relevant UPN policies in a DR like fashion.
  arXiv  Detail & Related papers  (2022-02-11T18:27:23Z)
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs [71.47895794305883]
  We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning setting. We present an SPI for this RL setting that takes into account the preferences of the algorithm's user for handling the trade-offs for different reward signals.
  arXiv  Detail & Related papers  (2021-05-31T21:04:21Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.