Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak

• URL: http://arxiv.org/abs/2607.14147v1
• Date: Tue, 14 Jul 2026 09:30:34 GMT
• Title: Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak
• Abstract summary: Aligned language models refuse harmful requests, but a one-line prefill strips the refusal.<n>A linear probe reads harm as high as on the refused ones (0.91-0.98), while behavioral refusal drops to chance.<n>We localize it to an early window: a dose-matched position control shows the first half of the response suffices to break refusal, while the second half is nearly inert.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aligned language models refuse harmful requests, but a one-line prefill ("Sure, here is") strips the refusal. We ask where and how it fails. The harm representation stays intact: on the prompts the attack flips to compliance, a linear probe reads harm as high as on the refused ones (0.91-0.98), while behavioral refusal drops to chance. This holds across four models and three families (1.5-3.8B, and at 14B). Refusal is therefore a shallow, response-site computation. We localize it to an early window: a dose-matched position control shows the first half of the response suffices to break refusal, while the second half is nearly inert. Three causal probes converge on that window. Restoring the harm direction there partially re-engages refusal. Injecting the model's own refuse-state reverses the jailbreak (74%, held-out). And knocking out the early response's attention to the prefill, but not an equal attention mass elsewhere, selectively collapses the harmful continuation. A base-model control identifies the mechanism: the same knockout collapses the continuation prefill-specifically even in a non-safety-tuned base model (64% to 25% harmful content vs a matched control's 64%, replicated at 7B). So the prefill's grip is generic autoregressive conditioning, not safety-specific suppression, and "refusal restoration" is a model-dependent fallback. The dominant mechanism is passive. A small safety-specific attractor remains on top (logit-trace concentration 0.24 vs 0.03), whose active-vs-passive character we size but do not fully separate. No single direction or component is a clean handle either: the decision is decodable but distributed, and refusal tracks harm rather than scary surface. The consequence is structural: a monitor reading the untouched prompt-side representation is immune by construction, but only to response-site attacks. The mechanism is diffuse; the failure surface is local.

Related papers

• Refusal Reads Only a Slice of What the Model Knows: Harm-Keyed Routing and Its Exceptions Across Model Families [1.0829694003408499]
  We show that alignment applied after pretraining is shallow in a measurable way.<n>A single direction in a model's residual stream can be edited out, and the model stops refusing harmful requests.<n>We ask what it reads, and we separate that from what the model comprehends.
  arXiv  Detail & Related papers  (2026-09-13T19:49:44Z)
• How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE [9.309878989186581]
  Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream.<n>We study whether it survives the shift to frontier mixture-of-experts models whose residual streams are no longer a single tensor and whose weights ship quantized.
  arXiv  Detail & Related papers  (2026-09-09T06:48:08Z)
• Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks [1.0152838128195467]
  Refusal training protects AI models from jailbreaks by training models to decline unsafe queries, reducing the risk of misuse.<n>Recent work finds that refusal behavior in aligned language models can be mediated by a single activation direction or a low-dimensional refusal subspace shared across harmful prompts.<n>Yet it remains unclear why safety-critical features in a wide range of models emerge in a concentrated, low-dimensional structure.
  arXiv  Detail & Related papers  (2026-08-26T05:35:54Z)
• Broken Symmetry in LLM Refusal: Answer Release Is More Local Than Refusal Restoration [24.010665397925333]
  When a language model refuses to answer a prompt, it is unclear whether the correct answer is erased from its internal representations, or merely suppressed at the output layer.<n>We investigate this mechanism using a controlled withhold setting, which yields perfectly matched answering and refusal trajectories for bidirectional activation patching.
  arXiv  Detail & Related papers  (2026-08-16T14:47:33Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Can LLMs Reliably Self-Report Adversarial Prefills, and How? [9.80193616788089]
  We show that large language models (LLMs) exhibit introspective capability on benign tasks.<n>We examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack.
  arXiv  Detail & Related papers  (2026-06-22T17:56:30Z)
• Latent-space Attacks for Refusal Evasion in Language Models [14.290157825353846]
  We recast refusal suppression as a latent-space evasion attack against linear probes trained to separate refused from answered prompts.<n>We achieve state-of-the-art attack success rate across 15 instruction-tuned, multimodal, and reasoning models.
  arXiv  Detail & Related papers  (2026-05-20T20:10:27Z)
• Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts [51.84894623128418]
  Modern reasoning language models generate dense, sequential chain-of-thought traces implicitly assuming that every token contributes and that steps must be consumed in order.<n>We challenge both assumptions through a systematic intervention pipeline--removal, masking, shuffling, and noise injection--applied to model-generated reasoning chains.<n>We find that answer extraction operates on a sparse, order-insensitive, and structurally robust informational substrate.
  arXiv  Detail & Related papers  (2026-05-08T06:15:50Z)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away [97.11976870616273]
  We propose a lightweight inference-time defense that treats safety recovery as a satising constraint rather than an objective.<n>In our evaluations across six open-source MLRMs and four jailbreak benchmarks, SafeThink reduces attack success rates by 30-60%.
  arXiv  Detail & Related papers  (2026-02-11T18:09:17Z)
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning? [68.82210578851442]
  We investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens.<n>Using a linear probing approach to trace refusal intentions across token positions, we discover a phenomenon termed as textbfrefusal cliff<n>We propose textbfCliff-as-a-Judge, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment.
  arXiv  Detail & Related papers  (2025-10-07T15:32:59Z)
• An Embarrassingly Simple Defense Against LLM Abliteration Attacks [47.347413305965006]
  A recent attack, termed abliteration, identifies and suppresses the single latent direction most responsible for refusal behavior.<n>We propose a defense that fundamentally alters how models express refusal.<n>Fine-tuning Llama-2-7B-Chat and Qwen2.5-Instruct yields models that maintain high refusal rates under abliteration.
  arXiv  Detail & Related papers  (2025-05-25T09:18:24Z)
• Navigating the OverKill in Large Language Models [84.62340510027042]
  We investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to an over-attention of harmful words like 'kill' and prompts emphasizing safety will exacerbate overkill. We introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon.
  arXiv  Detail & Related papers  (2024-01-31T07:26:47Z)
• Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases [32.2246459413988]
  Red-teaming aims to jailbreak a model's safety behavior to make it act as a helpful agent disregarding the harmfulness of the query. We present a new perspective on safety research i.e., red-teaming through Unalignment. Unalignment tunes the model parameters to break model guardrails that are not deeply rooted in the model's behavior.
  arXiv  Detail & Related papers  (2023-10-22T13:55:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.