Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding

• URL: http://arxiv.org/abs/2405.18180v2
• Date: Fri, 31 Jan 2025 10:45:55 GMT
• Title: Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding
• Authors: Daniel Bethell, Simos Gerasimou, Radu Calinescu, Calum Imrie,
• Abstract summary: We introduce mboxADVICE (Adaptive Shielding with a Contrastive Autoencoder), a novel technique that distinguishes safe and unsafe features of state-action pairs during training. Our comprehensive experimental evaluation shows that ADVICE significantly reduces safety violations ($approx!!50%$) during training, with a competitive outcome reward compared to other techniques.
• Score: 5.5929450570003185
• License:
• Abstract: Empowering safe exploration of reinforcement learning (RL) agents during training is a critical challenge towards their deployment in many real-world scenarios. When prior knowledge of the domain or task is unavailable, training RL agents in unknown, \textit{black-box} environments presents an even greater safety risk. We introduce \mbox{ADVICE} (Adaptive Shielding with a Contrastive Autoencoder), a novel post-shielding technique that distinguishes safe and unsafe features of state-action pairs during training, and uses this knowledge to protect the RL agent from executing actions that yield likely hazardous outcomes. Our comprehensive experimental evaluation against state-of-the-art safe RL exploration techniques shows that ADVICE significantly reduces safety violations ($\approx\!\!50\%$) during training, with a competitive outcome reward compared to other techniques.

Related papers

• ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning [48.536695794883826]
  We present ActSafe, a novel model-based RL algorithm for safe and efficient exploration. We show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time. In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements.
  arXiv  Detail & Related papers  (2024-10-12T10:46:02Z)
• Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning [57.84059344739159]
  "Shielding" is a popular technique to enforce safety inReinforcement Learning (RL) We propose a new permissibility-based framework to deal with safety and shield construction.
  arXiv  Detail & Related papers  (2024-05-29T18:00:21Z)
• Provable Safe Reinforcement Learning with Binary Feedback [62.257383728544006]
  We consider the problem of provable safe RL when given access to an offline oracle providing binary feedback on the safety of state, action pairs. We provide a novel meta algorithm, SABRE, which can be applied to any MDP setting given access to a blackbox PAC RL algorithm for that setting.
  arXiv  Detail & Related papers  (2022-10-26T05:37:51Z)
• Guiding Safe Exploration with Weakest Preconditions [15.469452301122177]
  In reinforcement learning for safety-critical settings, it is desirable for the agent to obey safety constraints at all points in time. We present a novel neurosymbolic approach called SPICE to solve this safe exploration problem.
  arXiv  Detail & Related papers  (2022-09-28T14:58:41Z)
• On the Robustness of Safe Reinforcement Learning under Observational Perturbations [27.88525130218356]
  We show that baseline adversarial attack techniques for standard RL tasks are not always effective for safe RL. One interesting and counter-intuitive finding is that the maximum reward attack is strong, as it can both induce unsafe behaviors and make the attack stealthy by maintaining the reward. This work sheds light on the inherited connection between observational robustness and safety in RL and provides a pioneer work for future safe RL studies.
  arXiv  Detail & Related papers  (2022-05-29T15:25:03Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention [17.017957942831938]
  Current approaches for tackling safe learning in reinforcement learning (RL) lead to a trade-off between safe exploration and fulfilling the task. We introduce a new two-player framework for safe RL called Distributive Exploration Safety Training Algorithm (DESTA) Our approach uses a new two-player framework for safe RL called Distributive Exploration Safety Training Algorithm (DESTA)
  arXiv  Detail & Related papers  (2021-10-27T14:35:00Z)
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations [64.39401322671803]
  This paper explores the possibility of safe RL algorithms with zero training-time safety violations. We propose an algorithm, Co-trained Barrier Certificate for Safe RL (CRABS), which iteratively learns barrier certificates, dynamics models, and policies.
  arXiv  Detail & Related papers  (2021-08-04T04:59:05Z)
• Conservative Safety Critics for Exploration [120.73241848565449]
  We study the problem of safe exploration in reinforcement learning (RL) We learn a conservative safety estimate of environment states through a critic. We show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates.
  arXiv  Detail & Related papers  (2020-10-27T17:54:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.