Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal

• URL: http://arxiv.org/abs/2509.09708v2
• Date: Fri, 10 Oct 2025 07:46:26 GMT
• Title: Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal
• Abstract summary: Refusal on harmful prompts is a key safety behaviour in instruction-tuned large language models (LLMs)<n>We study two public instruction-tuned models, Gemma-2-2B-IT and LLaMA-3.1-8B-IT, using sparse autoencoders (SAEs) trained on residual-stream activations.<n>Given a harmful prompt, we search the SAE latent space for feature sets whose ablation flips the model from refusal to compliance, demonstrating causal influence and creating a jailbreak.
• Score: 40.3000670795883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Refusal on harmful prompts is a key safety behaviour in instruction-tuned large language models (LLMs), yet the internal causes of this behaviour remain poorly understood. We study two public instruction-tuned models, Gemma-2-2B-IT and LLaMA-3.1-8B-IT, using sparse autoencoders (SAEs) trained on residual-stream activations. Given a harmful prompt, we search the SAE latent space for feature sets whose ablation flips the model from refusal to compliance, demonstrating causal influence and creating a jailbreak. Our search proceeds in three stages: (1) Refusal Direction: find a refusal-mediating direction and collect SAE features near that direction; (2) Greedy Filtering: prune to a minimal set; and (3) Interaction Discovery: fit a factorization machine (FM) that captures nonlinear interactions among the remaining active features and the minimal set. This pipeline yields a broad set of jailbreak-critical features, offering insight into the mechanistic basis of refusal. Moreover, we find evidence of redundant features that remain dormant unless earlier features are suppressed. Our findings highlight the potential for fine-grained auditing and targeted intervention in safety behaviours by manipulating the interpretable latent space.

Related papers

• SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defense [77.70598852234149]
  Text-to-Video (T2V) generative models are vulnerable to jailbreak attacks in real-world deployment.<n>We propose SafeCA, a feature-level defense mechanism for safe cross-attention localization and regularization.<n> Experimental results show that SafeCA reduces the jailbreak success rate by about 20% on mainstream T2V models.
  arXiv  Detail & Related papers  (2026-08-11T14:01:24Z)
• From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability [53.681678236115836]
  We study when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM.<n>Our framework derives an upper bound on the base model's expected risk using four measurable quantities.<n>We show that the bound becomes non-vacuous on GPT-2 Small, Gemma-2B, and Llama-3-8B at practical sample sizes.
  arXiv  Detail & Related papers  (2026-06-16T18:28:23Z)
• Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs [20.987619302834478]
  Backdoor attacks in large language models are often treated as isolated trigger-response failures.<n>We identify a shared latent mechanism that can be detected, causally controlled, and suppressed.<n>Our results suggest that many backdoors rely on a transferable latent mechanism, enabling unified detection and mitigation.
  arXiv  Detail & Related papers  (2026-06-06T03:41:44Z)
• Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders [20.13794189010598]
  Preference learning in large language models relies on reward models as proxies for human judgment.<n>We analyze this instability at the representation level under three semantic-preserving perturbation types.<n>We propose two SAE-based instability mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-07T16:48:48Z)
• Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs [0.0]
  Openweight Large Language Models (LLMs) have democratized agentic AI, yet finetuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance.<n>This creates a risk where third-party models are incorporated without strong behavioral guarantees.<n>We show that poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption.
  arXiv  Detail & Related papers  (2026-03-02T22:01:08Z)
• Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process [66.38541693477181]
  We propose an unsupervised framework for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors.<n>By segmenting chain-of-thought traces into sentence-level'steps', we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking.<n>We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space.
  arXiv  Detail & Related papers  (2025-12-30T05:09:11Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• Understanding Refusal in Language Models with Sparse Autoencoders [27.212781538459588]
  We use sparse autoencoders to identify latent features that causally mediate refusal behaviors.<n>We intervene on refusal-related features to assess their influence on generation.<n>This enables a fine-grained inspection of how refusal manifests at the activation level.
  arXiv  Detail & Related papers  (2025-05-29T15:33:39Z)
• Wolf Hidden in Sheep's Conversations: Toward Harmless Data-Based Backdoor Attacks for Jailbreaking Large Language Models [69.11679786018206]
  Supervised fine-tuning (SFT) aligns large language models with human intent by training them on labeled task-specific data.<n>Recent studies have shown that malicious attackers can inject backdoors into these models by embedding triggers into the harmful question-answer pairs.<n>We propose a novel clean-data backdoor attack for jailbreaking LLMs.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders [3.6367827664262715]
  Large language models (LLMs) are now ubiquitous in user-facing applications, yet they still generate undesirable toxic outputs.<n>We leverage sparse autoencoders (SAEs) to identify toxicity-related directions in the residual stream of models and perform targeted activation steering.
  arXiv  Detail & Related papers  (2025-05-20T15:55:31Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender [73.09848497762667]
  We propose AdaSteer, an adaptive activation steering method that adjusts model behavior based on input characteristics.<n>AdaSteer steers input representations along both the Rejection Direction (RD) and Harmfulness Direction (HD)<n>Our results highlight the potential of interpretable model internals for real-time, flexible safety enforcement in LLMs.
  arXiv  Detail & Related papers  (2025-04-13T07:39:17Z)
• Feature-Aware Malicious Output Detection and Mitigation [8.378272216429954]
  We propose a feature-aware method for harmful response rejection (FMM)<n>FMM detects the presence of malicious features within the model's feature space and adaptively adjusts the model's rejection mechanism.<n> Experimental results demonstrate the effectiveness of our approach across multiple language models and diverse attack techniques.
  arXiv  Detail & Related papers  (2025-04-12T12:12:51Z)
• Don't Forget It! Conditional Sparse Autoencoder Clamping Works for Unlearning [0.306238659426286]
  Large Language Model (LLM) capabilities have brought great potential but also posed new risks.<n>For example, LLMs with knowledge of bioweapons, advanced chemistry, or cyberattacks could cause violence if placed in the wrong hands or during malfunctions.<n>Because of their nature as near-black boxes, intuitive interpretation of LLM internals remains an open research question.
  arXiv  Detail & Related papers  (2025-03-14T06:43:19Z)
• The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions [20.522881564776434]
  We find that safety-aligned behavior is jointly controlled by multi-dimensional directions.<n>By studying directions in the space, we first find that a dominant direction governs the model's refusal behavior.<n>We then measure how different directions promote or suppress the dominant direction.
  arXiv  Detail & Related papers  (2025-02-13T06:39:22Z)
• Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing [63.20133320524577]
  We show that editing a small subset of parameters can effectively modulate specific behaviors of large language models (LLMs)<n>Our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen.
  arXiv  Detail & Related papers  (2024-07-11T17:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.