Steering Language Model Refusal with Sparse Autoencoders

• URL: http://arxiv.org/abs/2411.11296v2
• Date: Thu, 22 May 2025 23:03:47 GMT
• Title: Steering Language Model Refusal with Sparse Autoencoders
• Authors: Kyle O'Brien, David Majercak, Xavier Fernandes, Richard Edgar, Blake Bullwinkel, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, Forough Poursabzi-Sangde,
• Abstract summary: This work uncovers a tension between SAE steering-based safety improvements and general model capabilities.<n>Our findings reveal important open questions about the nature of safety-relevant features in language models.
• Score: 16.304363931580273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Responsible deployment of language models requires mechanisms for refusing unsafe prompts while preserving model performance. While most approaches modify model weights through additional training, we explore an alternative: steering model activations at inference time via amplifying sparse autoencoder (SAE) features that mediate refusal. This work uncovers a fundamental tension between SAE steering-based safety improvements and general model capabilities. While feature steering successfully improves robustness against both single-turn and challenging multi-turn jailbreak attempts, we discover that this comes at a previously underexplored cost -- systematic degradation of performance across multiple benchmark tasks, even on safe inputs with no apparent connection to refusal behavior. This suggests that features mediating refusal may be more deeply entangled with general language model capabilities than previously understood. Our findings reveal important open questions about the nature of safety-relevant features in language models and the feasibility of isolating them for targeted intervention. While SAE-based steering shows promise as a flexible approach to enhancing language model safety, our results highlight the critical need to understand and address the mechanisms behind these capability tradeoffs before such techniques can be practically deployed.

Related papers

• SAFER: Probing Safety in Reward Models with Sparse Autoencoder [15.804171763844323]
  We present sparse Autoencoder For Enhanced Reward model (textbfSAFER)<n>We uncover human-interpretable features in reward model activations, enabling insight into safety-relevant decision-making.<n>Experiments show that SAFER can precisely degrade or enhance safety alignment with minimal data modification.
  arXiv  Detail & Related papers  (2025-07-01T11:04:03Z)
• Improved Representation Steering for Language Models [50.86411958644953]
  We show how to improve representation steering via our new Reference-free Preference Steering (RePS)<n>On Gemma models with sizes ranging from 2B to 27B, RePS outperforms all existing steering methods trained with a language modeling objective.<n>In suppression, RePS matches the language-modeling objective on Gemma-2 and outperforms it on the larger Gemma-3 variants.
  arXiv  Detail & Related papers  (2025-05-27T07:16:40Z)
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals [8.297367440457508]
  Recent studies on the safety alignment of large language models (LLMs) have revealed that existing approaches often operate superficially.<n>This paper identifies a fundamental cause of this superficiality: existing alignment approaches presume that models can implicitly learn a safety-related reasoning task during the alignment process.<n>By explicitly introducing a safety-related binary classification task and integrating its signals with our attention and decoding strategies, we eliminate this ambiguity.
  arXiv  Detail & Related papers  (2025-05-19T20:40:46Z)
• T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models [88.63040835652902]
  Text to video models are vulnerable to jailbreak attacks, where specially crafted prompts bypass safety mechanisms and lead to the generation of harmful or unsafe content.<n>We propose T2VShield, a comprehensive and model agnostic defense framework designed to protect text to video models from jailbreak threats.<n>Our method systematically analyzes the input, model, and output stages to identify the limitations of existing defenses.
  arXiv  Detail & Related papers  (2025-04-22T01:18:42Z)
• AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender [73.09848497762667]
  We propose AdaSteer, an adaptive activation steering method that adjusts model behavior based on input characteristics. AdaSteer steers input representations along both the Rejection Direction (RD) and Harmfulness Direction (HD) Our results highlight the potential of interpretable model internals for real-time, flexible safety enforcement in LLMs.
  arXiv  Detail & Related papers  (2025-04-13T07:39:17Z)
• Feature-Aware Malicious Output Detection and Mitigation [8.378272216429954]
  We propose a feature-aware method for harmful response rejection (FMM) FMM detects the presence of malicious features within the model's feature space and adaptively adjusts the model's rejection mechanism. Experimental results demonstrate the effectiveness of our approach across multiple language models and diverse attack techniques.
  arXiv  Detail & Related papers  (2025-04-12T12:12:51Z)
• Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge [0.0]
  Large Language Models (LLMs) have revolutionized artificial intelligence, driving advancements in machine translation, summarization, and conversational agents.<n>Recent studies indicate that LLMs remain vulnerable to adversarial attacks designed to elicit biased responses.<n>This work proposes a scalable benchmarking framework to evaluate LLM robustness against adversarial bias elicitation.
  arXiv  Detail & Related papers  (2025-04-10T16:00:59Z)
• Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretability [7.73472615056109]
  We investigate the safety of compressed models by examining the mechanisms of refusal.<n>We propose a lightweight, computationally efficient method to enhance the safety of compressed models without compromising their performance or utility.
  arXiv  Detail & Related papers  (2025-04-05T16:00:44Z)
• Activation Space Interventions Can Be Transferred Between Large Language Models [0.0]
  We show that safety interventions can be transferred between models through learned mappings of their shared activation spaces.<n>We demonstrate this approach on two well-established AI safety tasks: backdoor removal and refusal of harmful prompts.<n>We also propose a new task, textitcorrupted capabilities, where models are fine-tuned to embed knowledge tied to a backdoor.
  arXiv  Detail & Related papers  (2025-03-06T13:38:44Z)
• DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing [62.43110639295449]
  Large Language Models (LLMs) are widely applied in decision making, but their deployment is threatened by jailbreak attacks. Delman is a novel approach leveraging direct model editing for precise, dynamic protection against jailbreak attacks. Delman directly updates a minimal set of relevant parameters to neutralize harmful behaviors while preserving the model's utility.
  arXiv  Detail & Related papers  (2025-02-17T10:39:21Z)
• In-Context Experience Replay Facilitates Safety Red-Teaming of Text-to-Image Diffusion Models [104.94706600050557]
  Text-to-image (T2I) models have shown remarkable progress, but their potential to generate harmful content remains a critical concern in the ML community.<n>We propose ICER, a novel red-teaming framework that generates interpretable and semantic meaningful problematic prompts.<n>Our work provides crucial insights for developing more robust safety mechanisms in T2I systems.
  arXiv  Detail & Related papers  (2024-11-25T04:17:24Z)
• Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level [10.658844160259104]
  Large language models (LLMs) have demonstrated immense utility across various industries. As LLMs advance, the risk of harmful outputs increases due to incorrect or malicious instruction prompts. This paper examines the LLMs' capability to recognize harmful outputs, revealing and quantifying their proficiency in assessing the danger of previous tokens.
  arXiv  Detail & Related papers  (2024-10-09T12:09:30Z)
• Applying Refusal-Vector Ablation to Llama 3.1 70B Agents [0.0]
  In this study, we apply refusal-vector ablation to Llama 3.1 70B and implement a simple agent scaffolding to create an unrestricted agent. Our findings imply that these refusal-vector ablated models can successfully complete harmful tasks, such as bribing officials or crafting phishing attacks. Our results imply that safety fine-tuning in chat models does not generalize well to agentic behavior, as we find that Llama 3.1 Instruct models are willing to perform most harmful tasks without modifications.
  arXiv  Detail & Related papers  (2024-10-08T13:42:36Z)
• Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle [11.619942198710334]
  We present our methodology for safety aligning the Phi-3 series of language models. We utilize a "break-fix" cycle, performing multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification. Results indicate that this approach iteratively improved the performance of the Phi-3 models across a wide range of responsible AI benchmarks.
  arXiv  Detail & Related papers  (2024-07-18T18:06:59Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• Steering Without Side Effects: Improving Post-Deployment Control of Language Models [61.99293520621248]
  Language models (LMs) have been shown to behave unexpectedly post-deployment. We present KL-then-steer (KTS), a technique that decreases the side effects of steering while retaining its benefits. Our best method prevents 44% of jailbreak attacks compared to the original Llama-2-chat-7B model.
  arXiv  Detail & Related papers  (2024-06-21T01:37:39Z)
• DPP-Based Adversarial Prompt Searching for Lanugage Models [56.73828162194457]
  Auto-regressive Selective Replacement Ascent (ASRA) is a discrete optimization algorithm that selects prompts based on both quality and similarity with determinantal point process (DPP) Experimental results on six different pre-trained language models demonstrate the efficacy of ASRA for eliciting toxic content.
  arXiv  Detail & Related papers  (2024-03-01T05:28:06Z)
• Navigating the OverKill in Large Language Models [84.62340510027042]
  We investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to an over-attention of harmful words like 'kill' and prompts emphasizing safety will exacerbate overkill. We introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon.
  arXiv  Detail & Related papers  (2024-01-31T07:26:47Z)
• InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance [56.184255657175335]
  We develop textbfInferAligner, a novel inference-time alignment method that utilizes cross-model guidance for harmlessness alignment. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics. It significantly diminishes the Attack Success Rate (ASR) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks.
  arXiv  Detail & Related papers  (2024-01-20T10:41:03Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• SafeAMC: Adversarial training for robust modulation recognition models [53.391095789289736]
  In communication systems, there are many tasks, like modulation recognition, which rely on Deep Neural Networks (DNNs) models. These models have been shown to be susceptible to adversarial perturbations, namely imperceptible additive noise crafted to induce misclassification. We propose to use adversarial training, which consists of fine-tuning the model with adversarial perturbations, to increase the robustness of automatic modulation recognition models.
  arXiv  Detail & Related papers  (2021-05-28T11:29:04Z)
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts? [104.04999499189402]
  Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment. We propose an uncertainty-aware planning method, called emphrobust imitative planning (RIP) Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. We introduce an autonomous car novel-scene benchmark, textttCARNOVEL, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
  arXiv  Detail & Related papers  (2020-06-26T11:07:32Z)
• Online Parameter Estimation for Human Driver Behavior Prediction [5.927030511296174]
  We show that online parameter estimation applied to the Intelligent Driver Model captures nuanced individual driving behavior while providing collision free trajectories. We evaluate the closeness of our driver model to ground truth data demonstration and also assess the safety of the resulting emergent driving behavior.
  arXiv  Detail & Related papers  (2020-05-06T05:15:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.