When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models

• URL: http://arxiv.org/abs/2605.27851v1
• Date: Wed, 27 May 2026 02:09:30 GMT
• Title: When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models
• Abstract summary: Safety benchmark scores provide incomplete evidence of deployment readiness.<n> aligned language models often adhere to rigid rules even when a situational update flips which action is safe.<n>We test 12 models across a safety benchmark and two commonsense controls using paired variants where the nominally safe action produces harm.
• Score: 5.03712104249503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often adhere to rigid rules even when a situational update flips which action is safe. We term this failure brittle safety. To diagnose it, we introduce context-flip evaluation, testing 12 models across a safety benchmark (PacifAIst) and two commonsense controls using paired variants where the nominally safe action produces harm. Three findings emerge. First, brittle safety is safety-specific: all 12 models exhibit a safety-commonsense gap (mean +17.4 pp). Baseline accuracy fails to predict brittleness: among models above 90% baseline accuracy, brittleness rates range from 13.7% to 90.0%. Second, failures stem from policy override rather than miscomprehension: despite acknowledging the context change in every case, models persist via three distinct mechanisms that vary by update type and model family. Third, on a hand-audited probe of catastrophic consequence-flip scenarios, standard action-level guardrails catch none, while a state-aware validator catches all without false alarms on correct interventions. This indicates that action-level content moderation is systematically blind to consequence-flips, motivating state-aware architectural alternatives. We release our protocol, perturbed benchmarks, and deployment probe.

Related papers

• EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents [74.54929751174289]
  EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain.<n>It reduces average attack success rate from 45.6% to 10.0% at a 3.3-point utility cost.<n>It also achieves the best score on Agent-SafetyBench for every victim.
  arXiv  Detail & Related papers  (2026-09-05T05:56:41Z)
• Detecting Safety Training Modification in Language Models via Activation Analysis [0.0]
  We introduce AMS, a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space.<n>We validate AMS across 14 model configurations spanning 4 architecture families.<n>Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature.
  arXiv  Detail & Related papers  (2026-08-06T03:57:38Z)
• Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition [4.800831341795478]
  We use two Gemma-3-1B-IT finetunes on two complementary safety objectives: CARES harm-level classification and WildJailbreak adversarial refusal.<n>We merge the two fine-tunes using Linear, SLERP, TIES, and DARE-TIES, and evaluate the merged models on classification accuracy, attack resistance, and benign compliance.
  arXiv  Detail & Related papers  (2026-07-26T19:10:34Z)
• Evaluation Awareness Is Not One Capability: Evidence from Open Language Models [1.758143872501506]
  Safety benchmarks assume that test-condition behavior predicts deployment behavior, an assumption that fails if models detect evaluation cues and adapt.<n>This opens a gap between benchmark performance and deployment behavior.<n>We characterize this evaluation awareness through eight experiments across 37 open-weight models and seven families.
  arXiv  Detail & Related papers  (2026-06-22T16:48:43Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety [0.0]
  We report one of the largest controlled studies of scaffold effects on safety.<n>Map-reduce scaffolding degrades measured safety.<n>Switching from multiple-choice to open-ended format on identical items shifts safety scores by 5-20 percentage points.
  arXiv  Detail & Related papers  (2026-03-08T01:37:45Z)
• Trust The Typical [8.32740388004069]
  We introduce Trust The Typical (T3), a framework that operationalizes this principle by treating safety as an out-of-distribution (OOD) detection problem.<n>T3 learns the distribution of acceptable prompts in a semantic space and flags any significant deviation as a potential threat.<n>A single model trained only on safe English text transfers effectively to diverse domains and over 14 languages without retraining.
  arXiv  Detail & Related papers  (2026-02-04T14:06:46Z)
• When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals [0.07777489763207261]
  Current evaluations mostly report global rates such as false rejection or compliance.<n>We introduce "semantic confusion," a failure mode that captures such local inconsistency.<n>We propose three model-agnostic metrics at the token level: Confusion Index, Confusion Rate, and Confusion Depth.
  arXiv  Detail & Related papers  (2025-11-30T19:11:45Z)
• Qwen3Guard Technical Report [127.69960525219051]
  We present Qwen3Guard, a series of multilingual safety guardrail models with two specialized variants.<n>Generative Qwen3Guard casts safety classification as an instruction-following task to enable fine-grained tri-class judgments.<n>Stream Qwen3Guard introduces a token-level classification head for real-time safety monitoring.
  arXiv  Detail & Related papers  (2025-10-16T04:00:18Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)
• The Rogue Scalpel: Activation Steering Compromises LLM Safety [11.402179030703188]
  Activation steering is a technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference.<n>We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests.
  arXiv  Detail & Related papers  (2025-09-26T08:49:47Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.