Internalizing Safety Understanding in Large Reasoning Models via Verification

• URL: http://arxiv.org/abs/2605.08930v1
• Date: Sat, 09 May 2026 13:05:00 GMT
• Title: Internalizing Safety Understanding in Large Reasoning Models via Verification
• Abstract summary: We propose a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks.<n>We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks.
• Score: 33.2377930782685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While explicit Chain-of-Thought (CoT) empowers large reasoning models (LRMs), it enables the generation of riskier final answers. Current alignment paradigms primarily rely on externally enforced compliance, optimizing models to detect malicious prompts rather than evaluating the safety of their own outputs. We argue that this approach remains largely behavioral: our empirical analysis reveals that ostensibly aligned models lack intrinsic safety understanding, often failing to verify their own response safety and remaining vulnerable to adversarial jailbreaks. To address this fundamental limitation, we propose Safety Internal (SInternal), a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks to critique their own generated answers using expert reasoning trajectories. We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks. Furthermore, when combined with reinforcement learning, SInternal serves as a superior initialization compared to standard supervised fine-tuning, suggesting that internalizing safety understanding creates a more robust foundation for alignment than merely mimicking safe behaviors. Our codes are available at https://github.com/AlphaLab-USTC/SInternal

Related papers

• Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models [31.940554924036807]
  Large Reasoning Models (LRMs) remain highly vulnerable to sophisticated jailbreaks and direct harmful queries.<n>We employSupervised Fine-Tuning (SFT) to explicitly induce safe tags to trigger safety analysis and guidance.<n>We also apply Direct Preference Optimization (DPO) to further enhance the correctness and stability of the safety analysis and guidance.
  arXiv  Detail & Related papers  (2026-06-15T14:51:34Z)
• THINKSAFE: Self-Generated Safety Alignment for Reasoning Models [60.10077024249373]
  We propose ThinkSafe, a framework that restores safety alignment without external teachers.<n>Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm.<n> Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency.
  arXiv  Detail & Related papers  (2026-01-30T16:31:02Z)
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails [74.63933201261595]
  Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks.<n>LRMs remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks.<n>We propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps.
  arXiv  Detail & Related papers  (2025-10-24T09:32:25Z)
• Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models [93.5740266114488]
  Constructive Safety Alignment (CSA) protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results.<n>Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities.<n>We release Oy1, code, and the benchmark to support responsible, user-centered AI.
  arXiv  Detail & Related papers  (2025-09-02T03:04:27Z)
• UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases [57.69882799751655]
  We release UnsafeChain, a safety alignment dataset constructed from hard prompts with diverse sources.<n>We fine-tune three large reasoning models (LRMs) and compare them against recent SafeChain and STAR-1.<n>UnsafeChain consistently outperforms prior datasets, with even a 1K subset matching or surpassing baseline performance.
  arXiv  Detail & Related papers  (2025-07-29T10:08:52Z)
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning [21.399086197886202]
  Large language models (LLMs) possess latent safety understanding from their vast pretraining data.<n>We propose textbfAlphaAlign, a pure reinforcement learning framework with verifiable safety reward.<n>This allows the model to develop proactive safety reasoning capabilities without depending on supervised safety-specific reasoning data.
  arXiv  Detail & Related papers  (2025-07-20T14:47:03Z)
• ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning [64.32925552574115]
  ARMOR is a large language model that analyzes jailbreak strategies and extracts the core intent.<n> ARMOR achieves state-of-the-art safety performance, with an average harmful rate of 0.002 and an attack success rate of 0.06 against advanced optimization-based jailbreaks.
  arXiv  Detail & Related papers  (2025-07-14T09:05:54Z)
• Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning [76.56522719330911]
  Large Reasoning Models (LRMs) introduce a new generation paradigm of explicitly reasoning before answering.<n>LRMs pose great safety risks against harmful queries and adversarial attacks.<n>We propose SafeKey to better activate the safety aha moment in the key sentence.
  arXiv  Detail & Related papers  (2025-05-22T03:46:03Z)
• STAIR: Improving Safety Alignment with Introspective Reasoning [44.780098674618614]
  We propose STAIR, a framework that integrates SafeTy Alignment with Itrospective Reasoning.<n>We show that STAIR effectively mitigates harmful outputs while better preserving helpfulness, compared to instinctive alignment strategies.<n>With test-time scaling, STAIR achieves a safety performance comparable to Claude-3.5 against popular jailbreak attacks.
  arXiv  Detail & Related papers  (2025-02-04T15:02:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.