How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE

• URL: http://arxiv.org/abs/2609.09793v1
• Date: Wed, 09 Sep 2026 06:48:08 GMT
• Title: How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE
• Abstract summary: Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream.<n>We study whether it survives the shift to frontier mixture-of-experts models whose residual streams are no longer a single tensor and whose weights ship quantized.
• Score: 9.309878989186581
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hundred contrastive prompts, which makes it the canonical white-box attack on open-weight alignment. However, it has been established only on dense models up to roughly 70B parameters. We study whether it survives the shift to frontier mixture-of-experts (MoE) models whose residual streams are no longer a single tensor and whose weights ship quantized. We apply it to GLM-5.3-Flash (320B parameters, 288 routed experts, a four-wide hyper-connection residual, block-FP8). The attack survives the architecture, but what it reaches is no longer where a reader of the original recipe would look for it. Editing the attention, dense and routed-expert writers on their own removes 0.039, 0.016 and 0.148 of refusal respectively; editing all three together removes 0.776. As a result, 74% of the effect exists only under the joint intervention. The part the conventional recipe reaches by module-name matching accounts for 0.066 of that 0.776, which is why it fails silently on an MoE. The effect does not follow from removing just any direction: ablating a random direction orthogonal to it leaves refusal unchanged. A category-concentrated residue survives every edit we tried: subspaces fitted on violence, sexual content and hate leave measurable refusal at every rank from 1 to 12. We report the method, the 41-89 percentage-point reductions it achieves across seven harmful benchmarks with no detected change in capability, and the boundary where it stops.

Related papers

• The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards [1.667501390377158]
  Group-relative policy optimization can discard nocontrast rollout groups through dynamic sampling.<n>We identify and quantify a metricpredicate mismatch under composite shaped rewards.
  arXiv  Detail & Related papers  (2026-09-12T10:36:55Z)
• When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models [0.0]
  A language model, asked to verify 1,200 logical conclusions, answers YES every time.<n>linear probes on its hidden states read the correct verdict at 0.96 AUC.<n>We ask where the verdict is lost, and find the dominant failure is a single scalar.<n>The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset.
  arXiv  Detail & Related papers  (2026-09-04T00:26:45Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• When AI Writes, Who Gets Cited? Evidence of Citation Monoculture Across Language Models [45.91367912142692]
  We show that different models may still select the same narrow subset, producing citation monoculture without any single citation being wrong.<n>Eleven models from three vendors choose at most ten papers from uniformly random panels of thirty, with real titles and abstracts but fabricated authors, reassigned years, and hidden venues and citation counts.
  arXiv  Detail & Related papers  (2026-08-03T14:27:46Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak [0.0]
  Aligned language models refuse harmful requests, but a one-line prefill strips the refusal.<n>A linear probe reads harm as high as on the refused ones (0.91-0.98), while behavioral refusal drops to chance.<n>We localize it to an early window: a dose-matched position control shows the first half of the response suffices to break refusal, while the second half is nearly inert.
  arXiv  Detail & Related papers  (2026-07-14T09:30:34Z)
• Scaffold splits hide structural-frontier failures in ADMET models [25.408600489964726]
  We introduce a label-free structural-frontier split that reserves the sparsest and most physicochemically remote scaffold groups.<n>We evaluate it on six public experimental or curated ADMET tasks.
  arXiv  Detail & Related papers  (2026-07-12T12:25:50Z)
• Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain [0.0]
  Hate speech annotation pipelines collapse annotator disagreement into majority vote labels before training.<n>We show that this aggregation is not neutral.<n>Three downstream interventions of increasing sophistication all fail to recover boundary accuracy.
  arXiv  Detail & Related papers  (2026-06-27T06:56:07Z)
• Output Vector Editing for Memorization Mitigation in Large Language Models [68.30351930772788]
  Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks.<n>Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages.<n>We propose output vector editing, a constrained-optimization edit that locates a small set of neurons responsible for a memorized continuation.
  arXiv  Detail & Related papers  (2026-06-17T07:29:18Z)
• Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity [43.23157850664282]
  We introduce VOIR DIRE, a multimodal benchmark of 626 culturally paired image-prompt artifacts spanning U.S. and mainland Chinese contexts.<n>Across six MLLMs, the bias decomposes into two failures: a positivity-floor calibration failure and an orientation failure.<n>We recommend reporting alignment against each reference pool separately and treating cross-pool divergence as a judge property.
  arXiv  Detail & Related papers  (2026-06-12T15:53:40Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• An Embarrassingly Simple Defense Against LLM Abliteration Attacks [47.347413305965006]
  A recent attack, termed abliteration, identifies and suppresses the single latent direction most responsible for refusal behavior.<n>We propose a defense that fundamentally alters how models express refusal.<n>Fine-tuning Llama-2-7B-Chat and Qwen2.5-Instruct yields models that maintain high refusal rates under abliteration.
  arXiv  Detail & Related papers  (2025-05-25T09:18:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.