Explainably Safe Reinforcement Learning

• URL: http://arxiv.org/abs/2606.04634v1
• Date: Wed, 03 Jun 2026 09:06:52 GMT
• Title: Explainably Safe Reinforcement Learning
• Abstract summary: We propose a novel approach for explainable safe RL that enhances trust by providing human-interpretable explanations of the shield's decisions.<n>Our method represents the shielding policy as a hierarchy of decision trees, offering top-down, case-based explanations.
• Score: 3.5232085374661284
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Trust in a decision-making system requires both safety guarantees and the ability to interpret and understand its behavior. This is particularly important for learned systems, whose decision-making processes are often highly opaque. Shielding is a prominent model-based technique for enforcing safety in reinforcement learning. However, because shields are automatically synthesized using rigorous formal methods, their decisions are often similarly difficult for humans to interpret. Recently, decision trees became customary to represent controllers and policies. However, since shields are inherently non-deterministic, their decision tree representations become too large to be explainable in practice. To address this challenge, we propose a novel approach for explainable safe RL that enhances trust by providing human-interpretable explanations of the shield's decisions. Our method represents the shielding policy as a hierarchy of decision trees, offering top-down, case-based explanations. At design time, we use a world model to analyze the safety risks of executing actions in given states. Based on this analysis, we construct both the shield and a high-level decision tree that classifies states into risk categories (safe, critical, dangerous, unsafe), explaining why a situation may be safety-critical. At runtime, we generate localized decision trees that explain which actions are allowed and why others are deemed unsafe. Our method facilitates explainability of the safety aspect in safe-by-shielding reinforcement learning, requires no additional information beyond what is already used for shielding, incurs minimal overhead, and integrates readily into existing shielded RL pipelines. In our experiments, we compute explanations using decision trees that are several orders of magnitude smaller than the original shield.

Related papers

• Shielding for Higher-Order Safety [7.5385675686924865]
  Safety shields are runtime enforcement mechanisms that restrict the actions of a controller to guarantee safety.<n>This paper develops a finite-state safety-game construction for such high-order smoothness constraints.
  arXiv  Detail & Related papers  (2026-08-04T13:41:21Z)
• Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMs [13.624659633750861]
  Existing safety mechanisms for large language models (MLLMs) face a fundamental trade-off between safety and utility.<n>We propose a paradigm shift toward output-aware safety guardrails.<n>Our method operates within the model's hidden state space to predict whether the forthcoming generation will be unsafe.
  arXiv  Detail & Related papers  (2026-06-20T12:09:13Z)
• Internalizing Safety Understanding in Large Reasoning Models via Verification [33.2377930782685]
  We propose a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks.<n>We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks.
  arXiv  Detail & Related papers  (2026-05-09T13:05:00Z)
• Interval POMDP Shielding for Imperfect-Perception Agents [0.5729426778193399]
  Autonomous systems that rely on learned perception can make unsafe decisions when sensor readings are misclassified.<n>We study shielding: given a proposed action, a shield blocks actions that could violate safety.<n>Experiments show that our shielding approach improves the safety of the system over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-04-22T16:12:37Z)
• THINKSAFE: Self-Generated Safety Alignment for Reasoning Models [60.10077024249373]
  We propose ThinkSafe, a framework that restores safety alignment without external teachers.<n>Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm.<n> Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency.
  arXiv  Detail & Related papers  (2026-01-30T16:31:02Z)
• Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• Probabilistic Shielding for Safe Reinforcement Learning [51.35559820893218]
  In real-life scenarios, a Reinforcement Learning (RL) agent must often also behave in a safe manner, including at training time.<n>We present a new, scalable method, which enjoys strict formal guarantees for Safe RL.<n>We show that our approach provides a strict formal safety guarantee that the agent stays safe at training and test time.
  arXiv  Detail & Related papers  (2025-03-09T17:54:33Z)
• Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety [41.32331563680919]
  Large Language Models (LLMs) are vulnerable to jailbreak attacks that exploit weaknesses in traditional safety alignment.<n>We propose Reasoning-enhanced Finetuning for interpretable LLM Safety (Rational)<n>Rational trains models to engage in explicit safe reasoning before response.
  arXiv  Detail & Related papers  (2025-03-06T22:47:45Z)
• Safety Reasoning with Guidelines [63.15719512614899]
  Refusal Training (RT) struggles to generalize against various Out-of-Distribution (OOD) jailbreaking attacks.<n>We propose training model to perform safety reasoning for each query.
  arXiv  Detail & Related papers  (2025-02-06T13:01:44Z)
• SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior [56.10557932893919]
  We present SafetyAnalyst, a novel AI safety moderation framework.<n>Given an AI behavior, SafetyAnalyst uses chain-of-thought reasoning to analyze its potential consequences.<n>It aggregates effects into a harmfulness score using 28 fully interpretable weight parameters.
  arXiv  Detail & Related papers  (2024-10-22T03:38:37Z)
• Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints [15.904640266226023]
  We design a safety model that performs credit assignment to assess contributions of partial state-action trajectories on safety. We derive an effective algorithm for optimizing a safe policy using the learned safety model. We devise a method to dynamically adapt the tradeoff coefficient between safety reward and safety compliance.
  arXiv  Detail & Related papers  (2024-05-05T17:27:22Z)
• On Prompt-Driven Safeguarding for Large Language Models [172.13943777203377]
  We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO. Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness.
  arXiv  Detail & Related papers  (2024-01-31T17:28:24Z)
• Online Shielding for Reinforcement Learning [59.86192283565134]
  We propose an approach for online safety shielding of RL agents. During runtime, the shield analyses the safety of each available action. Based on this probability and a given threshold, the shield decides whether to block an action from the agent.
  arXiv  Detail & Related papers  (2022-12-04T16:00:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.