Related papers: Safe Exploration via Policy Priors

Safe Exploration via Policy Priors

URL: http://arxiv.org/abs/2601.19612v1
Date: Tue, 27 Jan 2026 13:45:28 GMT
Title: Safe Exploration via Policy Priors
Authors: Manuel Wendl, Yarden As, Manish Prajapat, Anton Pollak, Stelian Coros, Andreas Krause,
Abstract summary: We show that SOOPER guarantees safety throughout learning, and establish convergence to an optimal policy by bounding its cumulative regret.<n>Experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-art and validate our theoretical guarantees in practice.
Score: 45.58021831092113
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.g. simulated) environments. In this work, we tackle this challenge by utilizing suboptimal yet conservative policies (e.g., obtained from offline data or simulators) as priors. Our approach, SOOPER, uses probabilistic dynamics models to optimistically explore, yet pessimistically fall back to the conservative policy prior if needed. We prove that SOOPER guarantees safety throughout learning, and establish convergence to an optimal policy by bounding its cumulative regret. Extensive experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-art and validate our theoretical guarantees in practice.

Related papers

Vulnerability Analysis of Safe Reinforcement Learning via Inverse Constrained Reinforcement Learning [1.4707788677208018]
We propose an adversarial attack framework to reveal vulnerabilities of Safe RL policies.<n>Our framework learns a constraint model and a surrogate (learner) policy, enabling gradient-based attack optimization.
arXiv Detail & Related papers (2026-02-18T15:43:36Z)
Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
arXiv Detail & Related papers (2026-02-12T22:03:35Z)
Offline Safe Policy Optimization From Heterogeneous Feedback [35.454656807434006]
We introduce a framework that learns a policy based on pairwise preferences regarding the agent's behavior in terms of rewards, as well as binary labels indicating the safety of trajectory segments.<n>Our method successfully learns safe policies with high rewards, outperforming state-of-the-art baselines.
arXiv Detail & Related papers (2025-12-23T09:07:53Z)
A Provable Approach for End-to-End Safe Reinforcement Learning [17.17447653795906]
A longstanding goal in safe reinforcement learning (RL) is to ensure the safety of a policy throughout the entire process.<n>We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment.
arXiv Detail & Related papers (2025-05-28T00:48:20Z)
Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL [21.316556011382932]
Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints.<n>We propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL)<n>DRCORL first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference.
arXiv Detail & Related papers (2025-02-18T00:00:03Z)
ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning [48.536695794883826]
We present ActSafe, a novel model-based RL algorithm for safe and efficient exploration.<n>We show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time.<n>In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements.
arXiv Detail & Related papers (2024-10-12T10:46:02Z)
Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2022-07-04T11:43:23Z)
Conservative Safety Critics for Exploration [120.73241848565449]
We study the problem of safe exploration in reinforcement learning (RL) We learn a conservative safety estimate of environment states through a critic. We show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates.
arXiv Detail & Related papers (2020-10-27T17:54:25Z)
Cautious Reinforcement Learning with Logical Constraints [78.96597639789279]
An adaptive safe padding forces Reinforcement Learning (RL) to synthesise optimal control policies while ensuring safety during the learning process. Theoretical guarantees are available on the optimality of the synthesised policies and on the convergence of the learning algorithm.
arXiv Detail & Related papers (2020-02-26T00:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.