Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation

• URL: http://arxiv.org/abs/2510.18478v1
• Date: Tue, 21 Oct 2025 09:57:44 GMT
• Title: Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation
• Authors: Daniel Bethell, Simos Gerasimou, Radu Calinescu, Calum Imrie,
• Abstract summary: The Uncertain Safety Critic (USC) is a novel approach that integrates uncertainty-aware modulation and refinement into critic training.<n>USC reduces safety violations by approximately 40% while maintaining competitive or higher rewards.<n>It also reduces the error between predicted and true cost gradients by approximately 83%, breaking the prevailing trade-off between safety and performance.
• Score: 6.255435016547602
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Ensuring the safe exploration of reinforcement learning (RL) agents is critical for deployment in real-world systems. Yet existing approaches struggle to strike the right balance: methods that tightly enforce safety often cripple task performance, while those that prioritize reward leave safety constraints frequently violated, producing diffuse cost landscapes that flatten gradients and stall policy improvement. We introduce the Uncertain Safety Critic (USC), a novel approach that integrates uncertainty-aware modulation and refinement into critic training. By concentrating conservatism in uncertain and costly regions while preserving sharp gradients in safe areas, USC enables policies to achieve effective reward-safety trade-offs. Extensive experiments show that USC reduces safety violations by approximately 40% while maintaining competitive or higher rewards, and reduces the error between predicted and true cost gradients by approximately 83%, breaking the prevailing trade-off between safety and performance and paving the way for scalable safe RL.

Related papers

• Safety-Biased Policy Optimisation: Towards Hard-Constrained Reinforcement Learning via Trust Regions [7.419036996978718]
  Reinforcement learning (RL) in safety-critical domains requires agents to maximise rewards while strictly adhering to safety constraints.<n>We propose Safety-Biased Trust Region Policy optimisation (SB-TRPO), a new trust-region algorithm for hard-constrained RL.
  arXiv  Detail & Related papers  (2025-12-29T07:15:07Z)
• Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards [23.15178050525514]
  Safe Reinforcement Learning (Safe RL) aims to train an RL agent to maximize its performance in real-world environments while adhering to safety constraints.<n>We propose a novel safe RL approach called Safety Modulated Policy Optimization (SMPO), which enables safe policy function learning.
  arXiv  Detail & Related papers  (2025-04-03T21:35:22Z)
• Probabilistic Shielding for Safe Reinforcement Learning [51.35559820893218]
  In real-life scenarios, a Reinforcement Learning (RL) agent must often also behave in a safe manner, including at training time.<n>We present a new, scalable method, which enjoys strict formal guarantees for Safe RL.<n>We show that our approach provides a strict formal safety guarantee that the agent stays safe at training and test time.
  arXiv  Detail & Related papers  (2025-03-09T17:54:33Z)
• Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning [57.84059344739159]
  "Shielding" is a popular technique to enforce safety inReinforcement Learning (RL) We propose a new permissibility-based framework to deal with safety and shield construction.
  arXiv  Detail & Related papers  (2024-05-29T18:00:21Z)
• Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery [13.333197887318168]
  Safety is one of the main challenges in applying reinforcement learning to realistic environmental tasks. We propose a method to construct a boundary that discriminates safe and unsafe states. Our approach has better task performance with less safety violations than state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2023-06-24T12:02:50Z)
• Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments [84.3830478851369]
  We propose a safe reinforcement learning approach that can jointly learn the environment and optimize the control policy. Our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.
  arXiv  Detail & Related papers  (2022-09-29T20:49:25Z)
• Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
  We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-07-04T11:43:23Z)
• Enhancing Safe Exploration Using Safety State Augmentation [71.00929878212382]
  We tackle the problem of safe exploration in model-free reinforcement learning. We derive policies for scheduling the safety budget during training. We show that Simmer can stabilize training and improve the performance of safe RL with average constraints.
  arXiv  Detail & Related papers  (2022-06-06T15:23:07Z)
• Conservative Safety Critics for Exploration [120.73241848565449]
  We study the problem of safe exploration in reinforcement learning (RL) We learn a conservative safety estimate of environment states through a critic. We show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates.
  arXiv  Detail & Related papers  (2020-10-27T17:54:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.