Surgical Refusal Ablation: Disentangling Safety from Intelligence via Concept-Guided Spectral Cleaning

• URL: http://arxiv.org/abs/2601.08489v1
• Date: Tue, 13 Jan 2026 12:21:44 GMT
• Title: Surgical Refusal Ablation: Disentangling Safety from Intelligence via Concept-Guided Spectral Cleaning
• Abstract summary: Safety-aligned language models systematically refuse harmful requests.<n>We argue this degradation occurs because the raw vector is polysemantic.<n>We introduce Surgical Refusal Ablation to distill these steering directions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safety-aligned language models systematically refuse harmful requests. While activation steering can modulate refusal, ablating the raw "refusal vector" calculated from contrastive harmful and harmless prompts often causes collateral damage and distribution drift. We argue this degradation occurs because the raw vector is polysemantic, entangling the refusal signal with core capability circuits and linguistic style. We introduce Surgical Refusal Ablation (SRA) to distill these steering directions. SRA constructs a registry of independent Concept Atoms representing protected capabilities and stylistic confounds, then uses ridge-regularized spectral residualization to orthogonalize the refusal vector against these directions. This yields a clean refusal direction that targets refusal-relevant structure while minimizing disruption to the model's semantic geometry. Across five models (Qwen3-VL and Ministral series), SRA achieves deep refusal reduction (0-2%) with negligible perplexity impact on Wikitext-2 (mean delta PPL approx. 0.02) and minimal distribution drift. Notably, standard ablation on Qwen3-VL-4B induces severe drift (first-token KL = 2.088), whereas SRA maintains the original distribution (KL = 0.044) while achieving the same 0% refusal rate. Using teacher-forced perplexity on GSM8K and MBPP as a high-resolution capability proxy, we show SRA preserves math and code distributions. These results suggest that common "model damage" is often "Ghost Noise," defined as the spectral bleeding of the dirty refusal direction into capability subspaces.

Related papers

• Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defense [49.05879299987137]
  Scaling laws are usually read as a capability story: lower language-modeling loss yields more useful models.<n>We study a safety consequence of this mechanism in emphcross-session decomposition attacks, where benign-looking subqueries are asked across independent interactions and later recomposed toward a forbidden objective.<n>We formalize this setting as emphcompositional safety risk and prove a conditional risk-transfer bound.
  arXiv  Detail & Related papers  (2026-08-28T05:39:29Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP) [1.827510863075184]
  Grounding-Aware Sensitivity by Perturbation (GASP) is a span-level detector that scores each answer sentence by how strongly its likelihood depends on the retrieved evidence.<n>GASP holds the answer fixed and re-scores it under the full context, under no context, and with each chunk removed, then measures the log-likelihood drops and Jensen-Shannon divergences.<n>We evaluate GASP on three benchmarks (RAGTruth, TofuEval, RAGBench) with three instruction-tuned scorers from two model families.
  arXiv  Detail & Related papers  (2026-07-05T10:35:30Z)
• The Geometry of Harmful Intent: Training-Free Anomaly Detection via Angular Deviation in LLM Residual Streams [0.0]
  We present LatentBiopsy, a training-free method for detecting harmful prompts by analysing the geometry of residual-stream activations in large language models.<n>We evaluate two complete model triplets from the Qwen3.5-0.8B and Qwen2.5-0.5B families.<n>Across all six variants, LatentBiopsy achieves AUROC $geq$0.937 for harmful-vs-normative detection and AUROC = 1.000 for discriminating harmful from benign-aggressive prompts.
  arXiv  Detail & Related papers  (2026-03-28T21:19:58Z)
• Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models [71.11171576590134]
  Vision-language models can be easily induced by visual jailbreak attacks to generate harmful content.<n>We propose NullSteer, a null-space projected activation defense framework.<n>We show that NullSteer significantly reduces harmful outputs under various jailbreak attacks.
  arXiv  Detail & Related papers  (2026-03-23T15:23:23Z)
• Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs [0.0]
  Openweight Large Language Models (LLMs) have democratized agentic AI, yet finetuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance.<n>This creates a risk where third-party models are incorporated without strong behavioral guarantees.<n>We show that poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption.
  arXiv  Detail & Related papers  (2026-03-02T22:01:08Z)
• Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models [5.957171492626586]
  Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models.<n>We show that LoRA's vulnerability is fundamentally spectral.<n>Regularized Low-Rank Adaptation (RoRA) improves forgetting by increasing spectral strength.
  arXiv  Detail & Related papers  (2026-01-09T20:54:47Z)
• Differentiated Directional Intervention A Framework for Evading LLM Safety Alignment [7.145846466297704]
  Safety alignment instills in Large Language Models a capacity to refuse malicious requests.<n>Prior works have modeled this refusal mechanism as a single linear direction in the activation space.<n>We introduce Differentiated Bi-Directional Intervention (DBDI), a new white-box framework that precisely neutralizes the safety alignment at critical layer.
  arXiv  Detail & Related papers  (2025-11-10T08:52:34Z)
• The Rogue Scalpel: Activation Steering Compromises LLM Safety [11.402179030703188]
  Activation steering is a technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference.<n>We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests.
  arXiv  Detail & Related papers  (2025-09-26T08:49:47Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs [5.282422823698107]
  Large Language Models (LLMs) are aligned to meet ethical standards and safety requirements by training them to refuse answering harmful or unsafe prompts.<n>In this paper, we demonstrate how adversaries can exploit LLMs' alignment to implant bias, or enforce targeted censorship.
  arXiv  Detail & Related papers  (2025-08-28T00:30:25Z)
• SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs [57.880467106470775]
  Attackers can inject imperceptible perturbations into the training data, causing the model to generate malicious, attacker-controlled captions.<n>We propose Semantic Reward Defense (SRD), a reinforcement learning framework that mitigates backdoor behavior without prior knowledge of triggers.<n>SRD uses a Deep Q-Network to learn policies for applying discrete perturbations to sensitive image regions, aiming to disrupt the activation of malicious pathways.
  arXiv  Detail & Related papers  (2025-06-05T08:22:24Z)
• REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective [57.57786477441956]
  We propose an adaptive and semantic optimization problem over the population of responses.<n>Our objective doubles the attack success rate (ASR) on Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.
  arXiv  Detail & Related papers  (2025-02-24T15:34:48Z)
• Steering Without Side Effects: Improving Post-Deployment Control of Language Models [61.99293520621248]
  Language models (LMs) have been shown to behave unexpectedly post-deployment. We present KL-then-steer (KTS), a technique that decreases the side effects of steering while retaining its benefits. Our best method prevents 44% of jailbreak attacks compared to the original Llama-2-chat-7B model.
  arXiv  Detail & Related papers  (2024-06-21T01:37:39Z)
• Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals [52.123343364599094]
  adversarial attacks place carefully crafted perturbations on normal examples to fool deep neural networks (DNNs) We first empirically show that the features of either clean signals or adversarial perturbations are redundant and span in low-dimensional linear subspaces respectively with minimal overlap. This makes it possible for DNNs to learn a subspace where only features of clean signals exist while those of perturbations are discarded.
  arXiv  Detail & Related papers  (2024-03-24T14:35:44Z)
• Discriminator-Free Generative Adversarial Attack [87.71852388383242]
  Agenerative-based adversarial attacks can get rid of this limitation. ASymmetric Saliency-based Auto-Encoder (SSAE) generates the perturbations. The adversarial examples generated by SSAE not only make thewidely-used models collapse, but also achieves good visual quality.
  arXiv  Detail & Related papers  (2021-07-20T01:55:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.