Robust Probabilistic Shielding for Safe Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2605.10293v1
• Date: Mon, 11 May 2026 09:54:30 GMT
• Title: Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
• Abstract summary: In offline reinforcement learning, we learn policies from fixed datasets without environment interaction.<n>A technique called safe policy improvement (SPI) provides a performance guarantee: with high probability, the new policy outperforms a given baseline policy, which is assumed to be safe.<n>We integrate these paradigms by extending shielding to offline RL, relying solely on the available dataset and knowledge of safe and unsafe states.
• Score: 12.352902259774345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In offline reinforcement learning (RL), we learn policies from fixed datasets without environment interaction. The major challenges are to provide guarantees on the (1) performance and (2) safety of the resulting policy. A technique called safe policy improvement (SPI) provides a performance guarantee: with high probability, the new policy outperforms a given baseline policy, which is assumed to be safe. Orthogonally, in the context of safe RL, a shield provides a safety guarantee by restricting the action space to those actions that are provably safe with respect to a given safety-relevant model. We integrate these paradigms by extending shielding to offline RL, relying solely on the available dataset and knowledge of safe and unsafe states. Then, we shield the policy improvement steps, guaranteeing, with high probability, a safe policy. Experimental results demonstrate that shielded SPI outperforms its unshielded counterpart, improving both average and worst-case performance, particularly in low-data regimes.

Related papers

• SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning [15.190197265099748]
  Safety guarantees are a prerequisite to the deployment of reinforcement learning (RL) agents in safety-critical tasks.<n>Current approaches either do not provide formal guarantees or verify policy safety only a posteriori.<n>We propose a novel a priori approach to safe policy updates in continual RL by introducing the Rashomon set.<n>We show that one can provide formal, provable guarantees for arbitrary RL algorithms used to update a policy by projecting their updates onto the Rashomon set.
  arXiv  Detail & Related papers  (2026-04-10T16:09:39Z)
• Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
  Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
  arXiv  Detail & Related papers  (2026-02-12T22:03:35Z)
• A Provable Approach for End-to-End Safe Reinforcement Learning [17.17447653795906]
  A longstanding goal in safe reinforcement learning (RL) is to ensure the safety of a policy throughout the entire process.<n>We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment.
  arXiv  Detail & Related papers  (2025-05-28T00:48:20Z)
• Probabilistic Shielding for Safe Reinforcement Learning [51.35559820893218]
  In real-life scenarios, a Reinforcement Learning (RL) agent must often also behave in a safe manner, including at training time.<n>We present a new, scalable method, which enjoys strict formal guarantees for Safe RL.<n>We show that our approach provides a strict formal safety guarantee that the agent stays safe at training and test time.
  arXiv  Detail & Related papers  (2025-03-09T17:54:33Z)
• Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL [21.316556011382932]
  Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints.<n>We propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL)<n>DRCORL first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference.
  arXiv  Detail & Related papers  (2025-02-18T00:00:03Z)
• Verification-Guided Shielding for Deep Reinforcement Learning [4.418183967223081]
  Deep Reinforcement Learning (DRL) has emerged as an effective approach to solving real-world tasks. Various methods have been put forth to address this issue by providing formal safety guarantees. We present verification-guided shielding -- a novel approach that bridges the DRL reliability gap by integrating these two methods.
  arXiv  Detail & Related papers  (2024-06-10T17:44:59Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
  We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-07-04T11:43:23Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs [71.47895794305883]
  We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning setting. We present an SPI for this RL setting that takes into account the preferences of the algorithm's user for handling the trade-offs for different reward signals.
  arXiv  Detail & Related papers  (2021-05-31T21:04:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.