SafeDiffusion-R1: Online Reward Steering for Safe Diffusion Post-Training

• URL: http://arxiv.org/abs/2605.18719v1
• Date: Mon, 18 May 2026 17:50:04 GMT
• Title: SafeDiffusion-R1: Online Reward Steering for Safe Diffusion Post-Training
• Abstract summary: Diffusion models have been widely studied for removing unsafe content learned during pre-training.<n>Existing methods require expensive supervised data, either unsafe-text paired with safe-image groundtruth or negative/positive image pairs, making them impractical to scale.<n>We propose a novel online reinforcement learning framework that addresses both data scarcity and model degradation through post-training.
• Score: 47.04114541434947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have been widely studied for removing unsafe content learned during pre-training. Existing methods require expensive supervised data, either unsafe-text paired with safe-image groundtruth or negative/positive image pairs, making them impractical to scale. Furthermore, offline reinforcement learning and supervised fine-tuning approaches that generate synthetic data offline suffer from catastrophic forgetting, degrading generation quality. We propose a novel online reinforcement learning framework that addresses both data scarcity and model degradation through post-training with Group Relative Policy Optimization (GRPO) on both negative and positive text prompts. To eliminate the need for fine-tuning specialized safe/unsafe reward models, we introduce a \textit{steering reward mechanism} that exploits an inherent property of CLIP embeddings: steering text representations toward positive safety directions and away from negative ones in the embedding space. Our online-policy approach enables the model to learn from diverse prompts, including explicit unsafe content, without catastrophic forgetting. Extensive experiments demonstrate that our method reduces inappropriate content to 18.07\% (vs. 48.9\% for SD v1.4) and nudity detections to 15 (vs. 646 baseline) while improving compositional generation quality from 42.08\% to 47.83\% on GenEval. Remarkably, these safety gains generalize to out-of-domain unsafe prompts across seven harm categories, achieving state-of-the-art performance without supervised paired data or reward tuning. Github: https://github.com/MAXNORM8650/SafeDiffusion-R1.

Related papers

• SafeRedir: Prompt Embedding Redirection for Robust Unlearning in Image Generation Models [67.84174763413178]
  We introduce SafeRedir, a lightweight inference-time framework for robust unlearning via prompt embedding redirection.<n>We show that SafeRedir achieves effective unlearning capability, high semantic and perceptual preservation, robust image quality, and enhanced resistance to adversarial attacks.
  arXiv  Detail & Related papers  (2026-01-13T15:01:38Z)
• Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models [30.63803894651171]
  We propose Safe Text embedding Guidance (STG), a training-free approach to improve the safety of diffusion models.<n>STG adjusts the text embeddings based on a safety function evaluated on the expected final denoised image.<n>Experiments on various safety scenarios, including nudity, violence, and artist-style removal, show that STG consistently outperforms both training-based and training-free baselines.
  arXiv  Detail & Related papers  (2025-10-28T02:37:20Z)
• A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space [91.99501941169831]
  GuardSpace is a guardrail framework for preserving safety alignment throughout fine-tuning.<n>For Llama-2-7B-Chat fine-tuned on GSM8K, GuardSpace outperforms the state-of-the-art method AsFT.
  arXiv  Detail & Related papers  (2025-10-16T04:57:53Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• Safety Pretraining: Toward the Next Generation of Safe AI [68.99129474671282]
  We present a data-centric pretraining framework that builds safety into the model from the start.<n>Our framework consists of four key steps: Safety Filtering, Safety Rephrasing, Native Refusal and Harmfulness-Tag annotated pretraining.<n>Our safety-pretrained models reduce attack success rates from 38.8% to 8.4% on standard LLM safety benchmarks with no performance on general degradation tasks.
  arXiv  Detail & Related papers  (2025-04-23T17:58:08Z)
• POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization [36.27759448564185]
  Overgenerating completions for general-purpose prompts significantly boosts safety with only a minimal impact on usefulness.<n>Overgeneration for toxic prompts raises usefulness from 11.1% to 57.6% while preserving safety.<n>Applying POROVer increases usefulness further-from 57.6% to 82.1%-while keeping safety at comparable levels.
  arXiv  Detail & Related papers  (2024-10-16T19:56:22Z)
• Backtracking Improves Generation Safety [27.214507875399086]
  We propose backtracking, a technique that allows language models to "undo" and recover from their own unsafe generation. We show that models trained to backtrack are consistently safer than baseline models.
  arXiv  Detail & Related papers  (2024-09-22T20:28:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.