The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs

• URL: http://arxiv.org/abs/2606.18656v1
• Date: Wed, 17 Jun 2026 03:53:42 GMT
• Title: The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
• Abstract summary: We show that large language models (LLMs) may reject warranted conclusions even when they are explicitly supported by context.<n>We call this failure mode misfired alignment, where alignment-induced changes cause LLMs to override explicit evidence.<n>To quantify this phenomenon, we introduce VETO, a benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart.
• Score: 36.56059375552239
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment. Instead, this paper highlights the need for principled approaches to more advanced alignment. Alignment aims to ensure that large language models (LLMs) behave safely and reliably, including by avoiding unsafe inferences. However, we show that such safety-oriented behaviors can misfire: models may reject warranted conclusions even when they are explicitly supported by context. We call this failure mode misfired alignment, where alignment-induced changes cause LLMs to override explicit evidence. To quantify this phenomenon, specifically on stereotype-related alignment, we introduce VETO, a benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart. We benchmark 25 LLMs on VETO, and show that all LLMs, including the most recent ones, exhibit non-trivial (4.7 to 18.9%) MARs while all human participants achieve 0.0% MAR. Controlled priming experiments further show that alignment-induced cues can substantially amplify MAR across LLMs, indicating that these failures are not merely artifacts of individual examples but can be induced by safety-related framing. Mechanistic analyses on open-weight LLMs reveal late-layer suppression of evidence-supported answers, and comparisons between instruct and base LLMs suggest that this suppression emerges after instruction training. These findings show that current alignment methods can overgeneralize surface-level safety cues, to the point of overriding objective evidence, motivating more work on alignment objectives that better preserve contextual grounding.

Related papers

• Can LLMs Reliably Self-Report Adversarial Prefills, and How? [9.80193616788089]
  We show that large language models (LLMs) exhibit introspective capability on benign tasks.<n>We examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack.
  arXiv  Detail & Related papers  (2026-06-22T17:56:30Z)
• Right or Wrong, Models Comply: Directional Blindness in LLM Moral Judgment [13.021647720314016]
  We introduce Compliance Asymmetry, a bidirectional diagnostic that compares beneficial output change under helpful nudges with harmful nudges.<n>We find that this selectivity differs in factual and moral judgments.<n>Chain-of-thought prompting amplifies helpful and harmful compliance together, while identity-based prompting suppresses both by nearly identical margins.
  arXiv  Detail & Related papers  (2026-06-12T02:23:38Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• LLMs Can Unlearn Refusal with Only 1,000 Benign Samples [23.047329180544775]
  This study reveals a previously unexplored vulnerability in the safety alignment of Large Language Models.<n>Existing aligned LLMs predominantly respond to unsafe queries with refusals, which often begin with a fixed set of prefixes.<n>We introduce a novel textbfrefusal unlearning technique that exploits it.
  arXiv  Detail & Related papers  (2026-01-27T05:59:56Z)
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs [58.02809208460186]
  We revisit this paradox using high-quality traces from DeepSeek-R1 as demonstrations.<n>We find that adding more exemplars consistently degrades accuracy, even when demonstrations are optimal.<n>We introduce Insight-to-solve (I2S), a sequential test-time procedure that turns demonstrations into explicit, reusable insights.
  arXiv  Detail & Related papers  (2025-09-27T08:59:31Z)
• Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs [6.462219916993885]
  Alignment in large language models (LLMs) is used to enforce guidelines such as safety.<n>Yet, alignment fails in the face of jailbreak attacks that modify inputs to induce unsafe outputs.<n>We introduce and evaluate a new technique for jailbreak attacks.
  arXiv  Detail & Related papers  (2025-01-27T22:13:05Z)
• Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models [79.76293901420146]
  Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. Our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
  arXiv  Detail & Related papers  (2024-07-15T23:41:11Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.