RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoning

• URL: http://arxiv.org/abs/2602.04224v1
• Date: Wed, 04 Feb 2026 05:18:38 GMT
• Title: RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoning
• Abstract summary: Large Reasoning Models (LRMs) have achieved tremendous success with their chain-of-thought (CoT) reasoning.<n>LRMs often fail to generalize against diverse and complex jailbreak attacks.<n>We propose a Risk-Aware Preference Optimization (RAPO) framework that enables LRM to adaptively identify and address the safety risks.
• Score: 29.005437376731738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Reasoning Models (LRMs) have achieved tremendous success with their chain-of-thought (CoT) reasoning, yet also face safety issues similar to those of basic language models. In particular, while algorithms are designed to guide them to deliberately refuse harmful prompts with safe reasoning, this process often fails to generalize against diverse and complex jailbreak attacks. In this work, we attribute these failures to the generalization of the safe reasoning process, particularly their insufficiency against complex attack prompts. We provide both theoretical and empirical evidence to show the necessity of a more sufficient safe reasoning process to defend against advanced attack prompts. Building on this insight, we propose a Risk-Aware Preference Optimization (RAPO) framework that enables LRM to adaptively identify and address the safety risks with appropriate granularity in its thinking content. Extensive experiments demonstrate that RAPO successfully generalizes multiple LRMs' safe reasoning adaptively across diverse attack prompts whilst preserving general utility, contributing a robust alignment technique for LRM safety. Our code is available at https://github.com/weizeming/RAPO.

Related papers

• Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs [11.540170528812768]
  Large reasoning models (LRMs) achieve remarkable success on complex tasks but remain vulnerable to harmful prompts that induce unsafe outputs.<n>Recent methods align LRMs using direct refusals or safety rationales, yet often focus on prompt patterns rather than intrinsic attack mechanisms.<n>We propose AdvSafe, a dual-adversarial framework that enables LRMs to internalize unsafety knowledge by explicitly deconstructing adversarial mechanisms.
  arXiv  Detail & Related papers  (2026-08-10T12:40:02Z)
• Internalizing Safety Understanding in Large Reasoning Models via Verification [33.2377930782685]
  We propose a framework that internalizes safety specifications by training LRMs exclusively on safety verification tasks.<n>We demonstrate that learning to verify induces a strong generalization for response safety, significantly enhancing robustness against out-of-domain jailbreaks.
  arXiv  Detail & Related papers  (2026-05-09T13:05:00Z)
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails [74.63933201261595]
  Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks.<n>LRMs remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks.<n>We propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps.
  arXiv  Detail & Related papers  (2025-10-24T09:32:25Z)
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention [53.25106308403173]
  We show that existing methods overlook the unique significance of safe reasoning, undermining their trustworthiness and posing potential risks in applications if unsafe reasoning is accessible for and exploited by malicious users.<n>We propose Intervened Preference Optimization (IPO), an alignment method that enforces safe reasoning by substituting compliance steps with safety triggers and constructing pairs for preference learning with strong signals.
  arXiv  Detail & Related papers  (2025-09-29T07:41:09Z)
• FuSaR: A Fuzzification-Based Method for LRM Safety-Reasoning Balance [16.657840274027958]
  Large Reasoning Models (LRMs) have demonstrated impressive performance across various tasks due to their powerful reasoning capabilities.<n>We propose a novel method to improve the safety of LLMs without sacrificing their reasoning capability.
  arXiv  Detail & Related papers  (2025-08-18T12:54:16Z)
• ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments [18.198349215500183]
  ReasoningGuard injects timely safety aha moments to steer harmless while helpful reasoning processes.<n>Our approach outperforms seven existing safeguards, achieving state-of-the-art safety defenses.
  arXiv  Detail & Related papers  (2025-08-06T08:35:10Z)
• HauntAttack: When Attack Follows Reasoning as a Shadow [34.70760016394781]
  We introduce HauntAttack, a novel and general-purpose black-box adversarial attack framework.<n>We modify key reasoning conditions in existing questions with harmful instructions.<n>We observe an average attack success rate of 70%, achieving up to 12 percentage points of absolute improvement over the strongest prior baseline.
  arXiv  Detail & Related papers  (2025-06-08T07:45:48Z)
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs [83.11815479874447]
  We propose a novel jailbreak attack framework, inspired by cognitive decomposition and biases in human cognition.<n>We employ cognitive decomposition to reduce the complexity of malicious prompts and relevance bias to reorganize prompts.<n>We also introduce a ranking-based harmfulness evaluation metric that surpasses the traditional binary success-or-failure paradigm.
  arXiv  Detail & Related papers  (2025-05-03T05:28:11Z)
• SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models [50.34706204154244]
  Acquiring reasoning capabilities catastrophically degrades inherited safety alignment.<n>Certain scenarios suffer 25 times higher attack rates.<n>Despite tight reasoning-answer safety coupling, MLRMs demonstrate nascent self-correction.
  arXiv  Detail & Related papers  (2025-04-09T06:53:23Z)
• Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking [54.10710423370126]
  We propose Reasoning-to-Defend (R2D), a training paradigm that integrates a safety-aware reasoning mechanism into Large Language Models' generation process.<n>CPO enhances the model's perception of the safety status of given dialogues.<n>Experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances.
  arXiv  Detail & Related papers  (2025-02-18T15:48:46Z)
• Safety Reasoning with Guidelines [63.15719512614899]
  Refusal Training (RT) struggles to generalize against various Out-of-Distribution (OOD) jailbreaking attacks.<n>We propose training model to perform safety reasoning for each query.
  arXiv  Detail & Related papers  (2025-02-06T13:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.