Leveraging Analytic Gradients in Provably Safe Reinforcement Learning

• URL: http://arxiv.org/abs/2506.01665v2
• Date: Sun, 27 Jul 2025 13:02:11 GMT
• Title: Leveraging Analytic Gradients in Provably Safe Reinforcement Learning
• Authors: Tim Walter, Hannah Markgraf, Jonathan Külz, Matthias Althoff,
• Abstract summary: Provably safe reinforcement learning is an active field of research that aims to provide such guarantees using safeguards.<n>We develop the first effective safeguard for analytic gradient-based reinforcement learning.
• Score: 6.5301153208275675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The deployment of autonomous robots in safety-critical applications requires safety guarantees. Provably safe reinforcement learning is an active field of research that aims to provide such guarantees using safeguards. These safeguards should be integrated during training to reduce the sim-to-real gap. While there are several approaches for safeguarding sampling-based reinforcement learning, analytic gradient-based reinforcement learning often achieves superior performance from fewer environment interactions. However, there is no safeguarding approach for this learning paradigm yet. Our work addresses this gap by developing the first effective safeguard for analytic gradient-based reinforcement learning. We analyse existing, differentiable safeguards, adapt them through modified mappings and gradient formulations, and integrate them with a state-of-the-art learning algorithm and a differentiable simulation. Using numerical experiments on three control tasks, we evaluate how different safeguards affect learning. The results demonstrate safeguarded training without compromising performance.

Related papers

• Safely Learning Controlled Stochastic Dynamics [61.82896036131116]
  We introduce a method that ensures safe exploration and efficient estimation of system dynamics.<n>After training, the learned model enables predictions of the system's dynamics and permits safety verification of any given control.<n>We provide theoretical guarantees for safety and derive adaptive learning rates that improve with increasing Sobolev regularity of the true dynamics.
  arXiv  Detail & Related papers  (2025-06-03T11:17:07Z)
• Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing [12.986006070964772]
  Safety alignment is an essential research topic for real-world AI applications.<n>Our study first identified the difficulty of eliminating such vulnerabilities without sacrificing the model's helpfulness.<n>Our method could enhance the model's helpfulness while maintaining safety, thus improving the trade-off-front.
  arXiv  Detail & Related papers  (2025-02-04T09:31:54Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)
• Approximate Shielding of Atari Agents for Safe Exploration [83.55437924143615]
  We propose a principled algorithm for safe exploration based on the concept of shielding. We present preliminary results that show our approximate shielding algorithm effectively reduces the rate of safety violations.
  arXiv  Detail & Related papers  (2023-04-21T16:19:54Z)
• Adaptive Aggregation for Safety-Critical Control [3.1692938090731584]
  We propose an adaptive aggregation framework for safety-critical control. Our algorithm can achieve fewer safety violations while showing better data efficiency compared with several baselines.
  arXiv  Detail & Related papers  (2023-02-07T16:53:33Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions [35.9713619595494]
  Reinforcement Learning and continuous nonlinear control have been successfully deployed in multiple domains of complicated sequential decision-making tasks. Given the exploration nature of the learning process and the presence of model uncertainty, it is challenging to apply them to safety-critical control tasks. We propose a emphprovably efficient episodic safe learning framework for online control tasks.
  arXiv  Detail & Related papers  (2022-07-29T00:54:35Z)
• Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning [72.97229770329214]
  We introduce a general approach for seeking high dimensional non-linear optimization problems in which maintaining safety during learning is crucial. Our approach called LBSGD is based on applying a logarithmic barrier approximation with a carefully chosen step size. We demonstrate the effectiveness of our approach on minimizing violation in policy tasks in safe reinforcement learning.
  arXiv  Detail & Related papers  (2022-07-21T11:14:47Z)
• Barrier Certified Safety Learning Control: When Sum-of-Square Programming Meets Reinforcement Learning [0.0]
  This work adopts control barrier functions over reinforcement learning, and proposes a compensated algorithm to completely maintain safety. Compared to quadratic programming based reinforcement learning methods, our sum-of-squares programming based reinforcement learning has shown its superiority.
  arXiv  Detail & Related papers  (2022-06-16T04:38:50Z)
• Fail-Safe Adversarial Generative Imitation Learning [9.594432031144716]
  We propose a safety layer that enables a closed-form probability density/gradient of the safe generative continuous policy, end-to-end generative adversarial training, and worst-case safety guarantees. The safety layer maps all actions into a set of safe actions, and uses the change-of-variables formula plus additivity of measures for the density. In an experiment on real-world driver interaction data, we empirically demonstrate tractability, safety and imitation performance of our approach.
  arXiv  Detail & Related papers  (2022-03-03T13:03:06Z)
• Data Generation Method for Learning a Low-dimensional Safe Region in Safe Reinforcement Learning [9.903083270841638]
  Safe reinforcement learning aims to learn a control policy while ensuring that neither the system nor the environment gets damaged during the learning process. For implementing safe reinforcement learning on highly nonlinear and high-dimensional dynamical systems, one possible approach is to find a low-dimensional safe region via data-driven feature extraction methods. As the reliability of the learned safety estimates is data-dependent, we investigate in this work how different training data will affect the safe reinforcement learning approach.
  arXiv  Detail & Related papers  (2021-09-10T19:22:43Z)
• Neural Network Repair with Reachability Analysis [10.384532888747993]
  Safety is a critical concern for the next generation of autonomy that is likely to rely heavily on deep neural networks for perception and control. This research proposes a framework to repair unsafe DNNs in safety-critical systems with reachability analysis.
  arXiv  Detail & Related papers  (2021-08-09T17:56:51Z)
• Cautious Reinforcement Learning with Logical Constraints [78.96597639789279]
  An adaptive safe padding forces Reinforcement Learning (RL) to synthesise optimal control policies while ensuring safety during the learning process. Theoretical guarantees are available on the optimality of the synthesised policies and on the convergence of the learning algorithm.
  arXiv  Detail & Related papers  (2020-02-26T00:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.