A Granular Study of Safety Pretraining under Model Abliteration

• URL: http://arxiv.org/abs/2510.02768v1
• Date: Fri, 03 Oct 2025 07:01:45 GMT
• Title: A Granular Study of Safety Pretraining under Model Abliteration
• Abstract summary: We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
• Score: 64.24346997570275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-weight LLMs can be modified at inference time with simple activation edits, which raises a practical question for safety: do common safety interventions like refusal training or metatag training survive such edits? We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions, and conduct a controlled evaluation across a granular sequence of Safety Pretraining checkpoints for SmolLM2-1.7B, alongside widely used open baselines. For each of 20 systems, original and abliterated, we issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity on a small human-labeled subset. We also probe whether models can identify refusal in their own outputs. Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration, quantifies how judge selection influences evaluation outcomes, and outlines a practical protocol for integrating inference-time edits into safety assessments. Code: https://github.com/shashankskagnihotri/safety_pretraining.

Related papers

• Client-Side Probing of Deleted Ridge Statistics in Federated Unlearning [8.170162585703787]
  Federated unlearning aims to remove a client's data from a shared model without retraining from scratch.<n>Some efficient systems make deletion exact by storing compact, additive summaries of the training features.<n>We show that these broadcasts can also reveal the hidden summaries.
  arXiv  Detail & Related papers  (2026-09-03T20:59:09Z)
• RISA: Response Inspection and Selective Actions for Refusal Calibration in Large Language Models [19.938570872083652]
  Training-time alignment improves refusal behavior by updating model parameters with safety data, but requires additional computation and training.<n>Inferno-time alignment aims to modify LLM behavior during inference without updating the underlying model parameters.<n>We propose Response Inspection and Selective Actions (RISA), an inference-time framework that inspects the initial response and selectively corrects refusal errors without updating the base model.
  arXiv  Detail & Related papers  (2026-09-01T06:37:42Z)
• Detecting Safety Training Modification in Language Models via Activation Analysis [0.0]
  We introduce AMS, a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space.<n>We validate AMS across 14 model configurations spanning 4 architecture families.<n>Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature.
  arXiv  Detail & Related papers  (2026-08-06T03:57:38Z)
• Boundary-targeted Membership Inference Attacks on Safety Classifiers [70.20833439671131]
  Safety classifiers are trained on sensitive datasets including discussions of self-harm and mental health.<n>We introduce a new boundary-targeted selection strategy that identifies low confidence examples.<n>Experiments show that an adversary can recover 19% of the conversations a safety classifier flagged as indicating user distress, at a 5% false-positive rate.<n>This is $3.5$ times more than attacking using state-of-the-art MIA methods alone.
  arXiv  Detail & Related papers  (2026-05-21T12:05:22Z)
• Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check [32.82170313959032]
  We introduce a novel safety alignment approach called Answer-Then-Check.<n>Our method enables models to directly answer the question in their thought and then critically evaluate its safety.<n>We find that training on a small subset of just 500 examples can achieve comparable performance to using the full dataset.
  arXiv  Detail & Related papers  (2025-09-15T06:47:35Z)
• LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users [50.18141341939909]
  We describe a vulnerability in language models trained with user feedback.<n>A single user can persistently alter LM knowledge and behavior.<n>We show that this attack can be used to insert factual knowledge the model did not previously possess.
  arXiv  Detail & Related papers  (2025-07-03T17:55:40Z)
• Probing the Robustness of Large Language Models Safety to Latent Perturbations [30.16804362984161]
  Safety alignment is a key requirement for building reliable Artificial General Intelligence.<n>We observe that minor latent shifts can still trigger unsafe responses in aligned models.<n>We introduce Layer-wise Adversarial Patch Training(LAPT), a fine-tuning strategy that injects controlled perturbations into hidden representations during training.
  arXiv  Detail & Related papers  (2025-06-19T07:03:05Z)
• Wolf Hidden in Sheep's Conversations: Toward Harmless Data-Based Backdoor Attacks for Jailbreaking Large Language Models [81.44934796068495]
  Supervised fine-tuning (SFT) aligns large language models with human intent by training them on labeled task-specific data.<n>Malicious attackers can inject backdoors into these models by embedding triggers into the harmful question-answer (QA) pairs.<n>We propose a novel textitclean-data backdoor attack for jailbreaking LLMs.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Shape it Up! Restoring LLM Safety during Finetuning [66.46166656543761]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs [7.597770587484936]
  We introduce CARES (Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for evaluating medical large language models (LLMs) safety in healthcare.<n> CARES includes over 18,000 prompts spanning eight medical safety principles, four harm levels, and four prompting styles to simulate both malicious and benign use cases.<n>Our analysis reveals that many state-of-the-art LLMs remain vulnerable to jailbreaks that subtly rephrase harmful prompts, while also over-refusing safe but atypically phrased queries.
  arXiv  Detail & Related papers  (2025-05-16T16:25:51Z)
• Superficial Safety Alignment Hypothesis [15.215130286922564]
  We propose the Superficial Safety Alignment Hypothesis (SSAH), which posits that safety alignment teaches an otherwise unsafe model to choose the correct reasoning direction.<n>We identify four types of attribute-critical components: Safety Critical Unit (SCU), Utility Critical Unit (UCU), Complex Unit (CU) and Redundant Unit (RU)<n>Our findings show that freezing certain safety-critical components during fine-tuning allows the model to retain its safety attributes while adapting to new tasks.
  arXiv  Detail & Related papers  (2024-10-07T19:53:35Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position.<n>DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence.
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• BFClass: A Backdoor-free Text Classification Framework [21.762274809679692]
  We propose BFClass, a novel efficient backdoor-free training framework for text classification. The backbone of BFClass is a pre-trained discriminator that predicts whether each token in the corrupted input was replaced by a masked language model. Extensive experiments demonstrate that BFClass can identify all the triggers, remove 95% poisoned training samples with very limited false alarms, and achieve almost the same performance as the models trained on the benign training data.
  arXiv  Detail & Related papers  (2021-09-22T17:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.