GradShield: Alignment Preserving Finetuning

• URL: http://arxiv.org/abs/2605.14194v1
• Date: Wed, 13 May 2026 23:19:55 GMT
• Title: GradShield: Alignment Preserving Finetuning
• Abstract summary: GradShield is a principled filtering method that safeguards Large Language Models (LLMs) during finetuning.<n>It removes potentially harmful data by computing a Finetuning Implicit Harmfulness Score (FIHS) for each data point.<n>It consistently maintains an Attack Success Rate (ASR) below $6%$ while preserving utility performance.
• Score: 14.841149847814796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) pose a significant risk of safety misalignment after finetuning, as models can be compromised by both explicitly and implicitly harmful data. Even some seemingly benign data can inadvertently steer a model towards misaligned behaviors. To address this, we introduce GradShield, a principled filtering method that safeguards LLMs during finetuning by identifying and removing harmful data points before they corrupt the model's alignment. It removes potentially harmful data by computing a Finetuning Implicit Harmfulness Score (FIHS) for each data point and employs an adaptive thresholding algorithm. We apply GradShield to multiple utility fine-tuning tasks across varying levels of harmful data and evaluate the safety and utility performance of the resulting LLMs using various metrics. The results show that GradShield outperforms all baseline methods, consistently maintaining an Attack Success Rate (ASR) below $6\%$ while preserving utility performance.

Related papers

• DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning [21.57697098714205]
  Large language models (LLMs) suffer from degraded safety capabilities even when fine-tuned with benign datasets.<n>Existing methods for identifying safety-degrading samples in benign datasets suffer from high computational costs and significant noise issues.<n>We propose DataShield to efficiently and effectively identify potential safety-degrading samples.
  arXiv  Detail & Related papers  (2026-05-29T09:04:27Z)
• Token-level Data Selection for Safe LLM Fine-tuning [15.039068315115372]
  Fine-tuning large language models (LLMs) on custom datasets has become a standard approach for adapting these models to specific domains and applications.<n>Recent studies have shown that such fine-tuning can lead to significant degradation in the model's safety.<n>We propose a novel framework that quantifies the safety risk of each token by measuring the loss difference between a safety-degraded model and a utility-oriented model.
  arXiv  Detail & Related papers  (2026-03-01T16:52:05Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment [24.364891513019444]
  In this paper, we show that fine-tuning datasets often contain samples with safety-degrading features that are not easily identifiable on the surface.<n>We propose LARF, a Layer-Aware Representation Filtering method.<n> Experimental results demonstrate that LARF can effectively identify benign data with safety-degrading features.
  arXiv  Detail & Related papers  (2025-07-24T17:59:24Z)
• Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks [32.73803760326097]
  Finetuning-as-a-Service (F) allows users to customize Large Language Models (LLMs) using their own data.<n>Prior works attempt to mitigate this issue by first constructing safety-aligned model and then finetuning the model on user data.<n>We propose a Refusal-Teacher (Ref-Teacher)-guided finetuning framework.
  arXiv  Detail & Related papers  (2025-06-09T02:10:51Z)
• Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets [49.412887135146725]
  We propose Safe Delta, a safety-aware post-training defense method for large language models (LLMs)<n>Our approach consistently preserves safety while ensuring that the utility gain from benign datasets remains unaffected.
  arXiv  Detail & Related papers  (2025-05-17T15:01:07Z)
• Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation [86.05704141217036]
  Black-box finetuning is an emerging interface for adapting state-of-the-art language models to user needs. We introduce covert malicious finetuning, a method to compromise model safety via finetuning while evading detection.
  arXiv  Detail & Related papers  (2024-06-28T17:05:46Z)
• Learning and Forgetting Unsafe Examples in Large Language Models [41.115096910603086]
  Large language models (LLMs) learn from third-party custom finetuning data. We show that while aligned LLMs can readily learn unsafe content, they also tend to forget it more significantly when finetuned on safer content. We introduce the "ForgetFilter" algorithm, which filters unsafe data based on how strong the model's forgetting signal is for that data.
  arXiv  Detail & Related papers  (2023-12-20T03:18:50Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)
• Making Harmful Behaviors Unlearnable for Large Language Models [50.44915524846857]
  Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process.
  arXiv  Detail & Related papers  (2023-11-02T09:18:21Z)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! [88.90694413503614]
  We find that the safety alignment of LLMs can be compromised by fine-tuning. We jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples. We advocate for further research efforts toward reinforcing safety protocols for the custom fine-tuning of aligned LLMs.
  arXiv  Detail & Related papers  (2023-10-05T17:12:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.