LoRA is All You Need for Safety Alignment of Reasoning LLMs

• URL: http://arxiv.org/abs/2507.17075v3
• Date: Fri, 24 Oct 2025 05:12:21 GMT
• Title: LoRA is All You Need for Safety Alignment of Reasoning LLMs
• Authors: Yihao Xue, Baharan Mirzasoleiman,
• Abstract summary: We show that using LoRA for SFT on refusal datasets effectively aligns the model for safety without harming its reasoning capabilities.<n>This is because restricting the safety weight updates to a low-rank space minimizes the interference with the reasoning weights.<n>Our experiments show that this approach produces highly safe LLMs--with safety levels comparable to full-model fine-tuning--without compromising their reasoning abilities.
• Score: 30.616817385956754
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning LLMs have demonstrated remarkable breakthroughs in solving complex problems that were previously out of reach. To ensure LLMs do not assist with harmful requests, safety alignment fine-tuning is necessary in the post-training phase. However, safety alignment fine-tuning has recently been shown to significantly degrade reasoning abilities, a phenomenon known as the "Safety Tax". In this work, we show that using LoRA for SFT on refusal datasets effectively aligns the model for safety without harming its reasoning capabilities. This is because restricting the safety weight updates to a low-rank space minimizes the interference with the reasoning weights. Our extensive experiments across four benchmarks covering math, science, and coding show that this approach produces highly safe LLMs--with safety levels comparable to full-model fine-tuning--without compromising their reasoning abilities. Our ablation studies further identify three key factors in LoRA: (1) rank-$1$ updates are sufficient to achieve the best reasoning and safety performance, (2) the up projection layers are the most critical modules, with LoRA applied to them alone achieving even better results, and (3) middle layers are more effective than early or late layers. Together, these findings show that strong safety and reasoning can be achieved at minimal computational cost when updates are applied in the right places. Additionally, we observe that LoRA induces weight updates with smaller overlap with the initial weights compared to full-model fine-tuning. Finally, while our attempts to further reduce this overlap yield only modest improvements on some tasks, they highlight the potential of developing methods that more reliably optimize the reasoning-safety tradeoff.

Related papers

• THINKSAFE: Self-Generated Safety Alignment for Reasoning Models [60.10077024249373]
  We propose ThinkSafe, a framework that restores safety alignment without external teachers.<n>Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm.<n> Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency.
  arXiv  Detail & Related papers  (2026-01-30T16:31:02Z)
• Understanding and Preserving Safety in Fine-Tuned LLMs [20.821783178639063]
  Fine-tuning can substantially degrade safety alignment, even when the fine-tuning data is harmless.<n>We propose safety-preserving fine-tuning (SPF), a lightweight approach that explicitly removes gradient components conflicting with the low-rank safety subspace.<n> SPF consistently maintains downstream task performance and recovers nearly all pre-trained safety alignment, even under adversarial fine-tuning scenarios.
  arXiv  Detail & Related papers  (2026-01-15T07:33:13Z)
• S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner [1.6114012813668932]
  Adapting Large Language Models (LLMs) using parameter-efficient fine-tuning techniques such as LoRA has enabled powerful capabilities in LLM-based agents.<n>Existing safety-aware adaptation methods often require access to both base and instruction-tuned model checkpoints.<n>We propose S3LoRA, a lightweight, data-free, and model-independent framework that mitigates safety risks in LoRA-adapted models by inspecting only the fine-tuned weight updates.
  arXiv  Detail & Related papers  (2025-08-20T21:08:29Z)
• Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs [4.580092836731863]
  Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) enhances adaptability while reducing computational costs.<n>Existing safety alignment methods struggle to capture complex parameter shifts, leading to suboptimal safety-utility trade-offs.<n>We propose Safe Pruning LoRA (SPLoRA), a novel pruning-based approach that selectively removes LoRA layers that weaken safety alignment.
  arXiv  Detail & Related papers  (2025-06-21T14:59:54Z)
• Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency [17.57889200051214]
  Fine-tuning a general-purpose large language model (LLM) for a specific domain or task has become a routine procedure for ordinary users.<n>We consider this to be a critical failure mode of LLMs due to the widespread uptake of fine-tuning, combined with the benign nature of the "attack"<n>Our experiments expose surprising variance in the results of the safety evaluation, even when seemingly inconsequential changes are made to the fine-tuning setup.
  arXiv  Detail & Related papers  (2025-06-20T17:57:12Z)
• LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning [61.594212398272184]
  Low-Rank Extrapolation (LoX) improves robustness against benign and malicious fine-tuning attacks.<n>LoX leads to 11% to 54% absolute reductions in attack success rates.
  arXiv  Detail & Related papers  (2025-06-18T16:30:02Z)
• Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models [50.89022445197919]
  Large Audio Language Models (LALMs) have extended the capabilities of Large Language Models (LLMs)<n>Recent research has revealed that LALMs remain vulnerable to harmful queries due to insufficient safety-alignment.
  arXiv  Detail & Related papers  (2025-05-26T08:25:25Z)
• How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study [90.34190170330481]
  Large Reasoning Models (LRMs) have achieved remarkable success on reasoning-intensive tasks such as mathematics and programming.<n>However, their enhanced reasoning capabilities do not necessarily translate to improved safety performance.<n>We present a comprehensive empirical study on how to enhance the safety of LRMs through Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2025-05-21T11:45:29Z)
• SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation [41.91948079316541]
  Recent studies have raised concerns that LoRA fine-tuning could potentially compromise the safety alignment in large language models.<n>In this paper, we propose Safety-alignment preserved Low-Rank Adaptation (SaLoRA)<n>Unlike previous LoRA methods and their variants, SaLoRA enables targeted modifications to LLMs without disrupting their original alignments.<n>Our experiments show that SaLoRA outperforms various adapters-based approaches across various evaluation metrics in different fine-tuning tasks.
  arXiv  Detail & Related papers  (2025-01-03T11:34:28Z)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence [76.11938177294178]
  We study how Low-Rank Adaptation (LoRA) and full-finetuning change pre-trained models.<n>We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure.<n>We extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension.
  arXiv  Detail & Related papers  (2024-10-28T17:14:01Z)
• Superficial Safety Alignment Hypothesis [8.297367440457508]
  We propose the Superficial Safety Alignment Hypothesis (SSAH), which posits that safety alignment should teach an otherwise unsafe model to choose the correct reasoning direction. We identify four types of attribute-critical components in safety-aligned large language models (LLMs) Our findings show that freezing certain safety-critical components 7.5% during fine-tuning allows the model to retain its safety attributes while adapting to new tasks.
  arXiv  Detail & Related papers  (2024-10-07T19:53:35Z)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models [65.06446825020578]
  Safety alignment is crucial to ensure that large language models (LLMs) behave in ways that align with human preferences and prevent harmful actions during inference. We aim to measure the risks in finetuning LLMs through navigating the LLM safety landscape.
  arXiv  Detail & Related papers  (2024-05-27T17:31:56Z)
• Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models [51.20476412037321]
  We propose Safe LoRA, a simple one-liner patch to the original LoRA implementation by introducing the projection of LoRA weights from selected layers to the safety-aligned subspace.<n>Our experiments demonstrate that when fine-tuning on purely malicious data, Safe LoRA retains similar safety performance as the original aligned model.
  arXiv  Detail & Related papers  (2024-05-27T05:04:05Z)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! [88.90694413503614]
  We find that the safety alignment of LLMs can be compromised by fine-tuning. We jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples. We advocate for further research efforts toward reinforcing safety protocols for the custom fine-tuning of aligned LLMs.
  arXiv  Detail & Related papers  (2023-10-05T17:12:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.