Safe-RULE: Safe Reinforcement UnLEarning

• URL: http://arxiv.org/abs/2606.09559v1
• Date: Mon, 08 Jun 2026 14:33:40 GMT
• Title: Safe-RULE: Safe Reinforcement UnLEarning
• Abstract summary: offline safe reinforcement learning (Safe RL) enables policy learning without online interactions.<n>We propose safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data.
• Score: 0.22940141855172028
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.

Related papers

• Skill-based Safe Reinforcement Learning with Risk Planning [23.15178050525514]
  Safe Reinforcement Learning (Safe RL) aims to ensure safety when an RL agent conducts learning by interacting with real-world environments.<n>We propose a novel Safe Skill Planning (SSkP) approach to enhance effective safe RL by exploiting auxiliary offline demonstration data.
  arXiv  Detail & Related papers  (2025-05-02T22:48:27Z)
• Safety Pretraining: Toward the Next Generation of Safe AI [68.99129474671282]
  We present a data-centric pretraining framework that builds safety into the model from the start.<n>Our framework consists of four key steps: Safety Filtering, Safety Rephrasing, Native Refusal and Harmfulness-Tag annotated pretraining.<n>Our safety-pretrained models reduce attack success rates from 38.8% to 8.4% on standard LLM safety benchmarks with no performance on general degradation tasks.
  arXiv  Detail & Related papers  (2025-04-23T17:58:08Z)
• Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL [21.316556011382932]
  Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints.<n>We propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL)<n>DRCORL first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference.
  arXiv  Detail & Related papers  (2025-02-18T00:00:03Z)
• ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning [48.536695794883826]
  We present ActSafe, a novel model-based RL algorithm for safe and efficient exploration.<n>We show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time.<n>In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements.
  arXiv  Detail & Related papers  (2024-10-12T10:46:02Z)
• FOSP: Fine-tuning Offline Safe Policy through World Models [3.7971075341023526]
  Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration.<n>In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy.
  arXiv  Detail & Related papers  (2024-07-06T03:22:57Z)
• Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding [6.255435016547602]
  We introduce ADVICE (Adaptive Shielding with a Contrastive Autoencoder), a novel technique that distinguishes safe and unsafe features of state-action pairs during training.<n>Our comprehensive experimental evaluation shows that ADVICE significantly reduces safety violations (approx 50%) during training, with a competitive outcome reward compared to other techniques.
  arXiv  Detail & Related papers  (2024-05-28T13:47:21Z)
• Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration [75.51109230296568]
  We argue that extracting expert policy from offline data to guide online exploration is a promising solution to mitigate the conserveness issue. We propose Guided Online Distillation (GOLD), an offline-to-online safe RL framework. GOLD distills an offline DT policy into a lightweight policy network through guided online safe RL training, which outperforms both the offline DT policy and online safe RL algorithms.
  arXiv  Detail & Related papers  (2023-09-18T00:22:59Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Safe Reinforcement Learning via Confidence-Based Filters [78.39359694273575]
  We develop a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard reinforcement learning techniques. We provide formal safety guarantees, and empirically demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-07-04T11:43:23Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.